14 ms·

New storage classes for Google Cloud Storage

215 points by AndrewDucker 10y ago

questionr 10y ago
Google is making their cloud offering very attractive compared to AWS. The issue with Glacier is the convoluted retrieval pricing. I understand they want to dissuade people from using it as a primary storage, but the potential for a surprise bill is hard to swallow. Interesting how they still offer unlimited storage through their consumer Amazon Drive service.
- Gigablah 10y ago
  The AWS web console is a pain point. I had a particularly frustrating experience once where attempting to restore a Glacier storage class file in S3 silently failed due to permission issues (I had forgotten to assign permissions to the bucket owner in my automated upload script), but the console told me that the file would be "available in 3-5 hours". I wasted days due to this. Meanwhile other competitors (e.g. Nearline) promise to have your file available within seconds...
  - boulos 10y ago
    Yep. Nearline is great. Just so we're clear though: milliseconds ;). There's no latency penalty to get your bytes back, even with Coldline. We do charge a retrieval fee because the pricing is determined by a "promise" that you won't touch the data frequently (roughly once a month for Nearline and once a quarter for Coldline), otherwise you should use our regular flavors. Disclosure: I work on Google Cloud and am a happy GCS customer.
- user5994461 10y ago
  The issue is not limited to Glacier. The "Infrequent Access" is a nightmare as well. Paying for storage, read and writes per GB and per request, rounded to 128kB blocks. WTF.
user5994461 10y ago
Summary: Google Cloud Storage is now (significantly) cheaper, faster and more available than Amazon S3. Also, it supports multi-region whereas S3 does not. Love the direction Google Cloud is taking <3
- Touche 10y ago
  The best thing about Google Cloud is the Console. I haven't seen a comparable console for managed services. Think about the UX nightmare that is the AWS Management Console and take that complexity and separate it out neatly and make everything available in a fast search box and you have Google Cloud Console. It's the primary reason I've switched over to Datastore and App Engine.
  - RhodesianHunter 10y ago
    Hmm. I actually really like the AWS management console. I find that everything is logically organized and easy to manipulate. That could just be because I've been using it for a long time.
    - rstupek 10y ago
      I agree with you. Google's console is neigh unusable
- tehbeard 10y ago
  What about ~~reliability~~ durability of storage (bitrot, not the uptime)? I didn't see any information comparing this, is it practically the same across both? edit: durability, not reliability.
  - shshhdhs 10y ago
    I think that you mean durability
  - klodolph 10y ago
    At this point, durability is hard to compare because the published numbers are total fantasies. S3's published durability is 11 9s, which I've complained about in the past because the chance that the human race will be wiped out next year by an asteroid impact is probably not too far from one in a hundred million, which puts a hard cap on durability at around eight nines. Then ask yourself, "is the chance that someone made a mistake in designing the system which causes catastrophic data loss higher or lower than one in a billion?" Yes, the number is total fantasy. The only actual meaning of the durability SLA is that if the durability SLA is violated, your cloud provider will give you some service credits. If the data is worth more than some service credits, this is small consolation. In practice, if you absolutely must compare durability, what you are trying to do is compare the frequency of black swan events, but actually estimating the frequency of those events requires access to proprietary data which you don't have. Both providers are undoubtedly storing the data in multiple locations on multiple types of storage systems with multiple layers of error coding. After thinking about this issue earlier, the most likely data loss scenario in my opinion was "I get injured, the company hires someone incompetent who doesn't pay the bill for a year while I'm in the hospital."
vbezhenar 10y ago
Coldline seems to be a perfect solution for personal backups. I used amazon glacier, but got hit with extremely huge bill, when I decided to retrieve my backups, so I'm not going to deal with them anymore, their retrieval pricing is absurd. If there's good UI and scripting solutions for manual and automatic backup into google coldline, I'm in.
- matt4077 10y ago
  You can just user their command line tool, it's got an rsync mode: gsutil -m rsync -P -d -r -x "node_modules|tmp|..."-e ~/ gs://you-will-spend-too-much-time-thinking-of-a-clever-bucket-name (although I should expand it a bit to better handle ignoring stuff by parsing .gitignore)
  - jon-wood 10y ago
    > you-will-spend-too-much-time-thinking-of-a-clever-bucket-name FQDNs to the rescue, I very much doubt anyone but me is going to call a bucket backups.ninjagiraffes.co.uk (and if someone does so now, well, good trolling)
    - dekhn 10y ago
      https://cloud.google.com/storage/docs/domain-name-verification https://cloud.google.com/storage/docs/domain-name-verificati... if you own a domain, only people who pass domain ownership verification can make buckets with that domain suffix. Combined with https://cloud.google.com/storage/docs/hosting-static-website https://cloud.google.com/storage/docs/hosting-static-website it means you can easily host static websites by just populating appropriately named buckets with your domain suffix. I think this is a very cool feature.
      - nsheridan 10y ago
        I really wish AWS would introduce domain verification for 'domain-shaped' bucket names.
- 10y ago
meirelles 10y ago
Google finally is taking a step over the cloud offerings. I was skeptical about them ~1.5 year ago, they where only following the AWS without big innovations.
Heliosmaster 10y ago
Talking about cold storage, i'd like to Pitch C14 of Online.net (the company behind Scaleway): https://www.online.net/en/c14 https://www.online.net/en/c14 0.2ct per Gb.
plandis 10y ago
Multi region support is interesting, but haven't most of GCPs outages affected all of their regions?
- hurstdog 10y ago
  Checking specifically for GCS at https://status.cloud.google.com/summary https://status.cloud.google.com/summary ... There are two outages there, one that only affected a few projects and one that affected only service in the central US (we have regions over much of the world). Anecdotally, from watching and working with other services internally I don't think most of our outages affect all regions. We actually spend a significant amount of engineering effort ensuring that we're as decoupled as possible. Disclaimer: I'm an Engineering Manager on Google Cloud Storage.
  - jonathanoliver 10y ago
    Is there some documentation anywhere talks about how Google Cloud (any product) creates isolation between various regions while at the same time exposing a simple "regionless" programming model?
    - hurstdog 10y ago
      Probably the best reference is the SRE book, specifically the chapters on loadbalancing and distributed consensus protocols. Other than that, the general approach is to minimize global control planes and dependencies in our software stack. In the case of GCS, we do have a single namespace which means we need to look up the locations of data early in the request. Once we know the locations of data we can route the request to the right datacenter to serve it. That global location table is highly replicated and cached, of course. When outages happen, most are caused by changes to the stack, so we also are careful to roll out code or configuration slowly and carefully, slowly increasing the blast radius after it's been proven safe. For example rolling out new binaries first to a few canary instances in one zone, then to a few instances in many regions, then to a full region, then to the world, all spread over a few days. Disclaimer: I'm an Engineering Manager on Google Cloud Storage.
andybak 10y ago
Dear god, the copy on that page is awful. The first paragraph is just vacuous twaddle. Anyone who starts a minor product announcement with the words "Today, we’re excited to announce..." deserves their own circle of hell. Most of that first section is redundant preamble and should be rewritten as bullet points. In fact the whole thing could do with being rewritten in a simple bullet-pointed skimmable form. That way I wouldn't have to come to HN comments to find out what the hell it's all about. Is it too early in the day for a drink?
- devopsproject 10y ago
  Try some coffee with a light breakfast, then move on to the harder stuff if you still can't cope with "words".
  - andybak 10y ago
    A couple of points: 1. It was 4 in the afternoon here when I posted that - a touch too late for breakfast even if it was a bit early for a drink 2. I quite like words when they are strung together well ;-) I still would like to hear someone defend that as a piece of marketing copy. I've re-read it and stand by my earlier diatribe.
    - devopsproject 10y ago
      You've wasted a lot of time on something that is easily ignored. Its time to move on.
      - deleted 10y ago
        [deleted]
      - andybak 10y ago
        And I feel better for it.
brianwawok 10y ago
So what happened to "Durable, reduced availability". It does not seem directly mapped into the new system. I guess it roughly maps to "regional"???
- cobookman 10y ago
  Yep. Regional is the equivalent of Durable reduced availability.
  - brianwawok 10y ago
    Doesn't look like things got auto migrated.. (whereas for the other cases they did get auto migrated)
AdmiralAsshat 10y ago
Cold storage looks cheap at first glance, but for the amount of data I have to backup (about 3 TB), I work out the cost to: (0.7 * / 100) * 3000 = $21 / month * 12 = $252 / yr For that price, I'm better off just paying $60 / yr for CrashPlan, which is unlimited storage. I'd say that applies to anyone who has a terabyte or more of stuff to back up.
- devopsproject 10y ago
  Have you tried downloading any significant portion of your backup from CrashPlan? Is so, how was performance?
  - AdmiralAsshat 10y ago
    Not a significant portion, no. CrashPlan is only my third layer of backup, the offsite backup. Each of my drives have a clone backup drive (typically just a WD My Passport that makes them easy to transport when I travel), and then a larger external that holds backups of all of them--I usually keep that one in storage and only plug it in once a month or so to sync. CP is the third layer of backup. The most I do in a typical week is use the Android app to grab a few movies or documents from the CP storage to get it onto my phone and tablet. I should probably "test" doing a recovery from CrashPlan at some point, but I fear that even getting a third of it (1TB) would probably trip some sort of overcharge from my ISP.
  - jrnichols 10y ago
    I have. It was decent. accidentally hosed a vmware image and it was easier for me to just pull the whole thing down from Crashplan. Speeds were decent, limited mostly by my cable modem. I don't recall the exact throughput but I remember looking and saying "Hey, this isn't bad" and not being annoyed by anything. Plus the price isn't too bad.
- crudbug 10y ago
  https://www.backblaze.com https://www.backblaze.com
kozikow 10y ago
I had to move to AWS entirely due to lack of GPU instances on google cloud. If someone from GCE is reading this - please add GPU instances. Yes, I know unrelated, but I like GCS much more than S3, and I can't use it since by "network" effect caused by GPUs eveyrthing have to run on AWS.
- boulos 10y ago
  Please ping me with your project id, and your need (e.g., "I want to run TensorFlow on 8 GPUs per VM, 4 VMs at a time, somewhere in Europe"). Disclosure: I work for Google Cloud (contact info in profile).
rsync 10y ago
In case it's not widely known, rsync.net maintains 'gsutil' in the environment (along with s3cmd) so you can move data between google cloud services and rsync.net. ssh user@rsync.net gsutil cp mscdex.exe gs://my-bucket ssh user@rsync.net gsutil rsync -d data gs://mybucket/data Although it should be noted that, circa 2016, the cool kids are all backing up to rsync.net with borg[1][2] (the "holy grail of backup software") which limits the use cases of gsutil with an rsync.net account. HN Readers discount. Just email and ask.[3] [1] https://borgbackup.readthedocs.io/en/stable/ https://borgbackup.readthedocs.io/en/stable/ [2] https://www.stavros.io/posts/holy-grail-backups/ https://www.stavros.io/posts/holy-grail-backups/ [3] info@rsync.net
- Veratyr 10y ago
  I say this most times I see you advertising and I'll say it again: You really need to work on your pricing to be competitive with things like this. Even with the HN discount your service is expensive and even with the Borg/Attic pricing which is cheaper still, your service is 3x more expensive than alternatives like Nearline (aside from bandwidth), which themselves can be used as a backend for Attic (just mount GCS locally). I really like the look of the service but I can't justify paying over 3x for it.
  - rsync 10y ago
    "your service is 3x more expensive than alternatives like Nearline (aside from bandwidth), which themselves can be used as a backend for Attic (just mount GCS locally)." Well, our storage platform is online and random access so it would be inappropriate to compare it to either nearline or glacier. The appropriate comparison is to S3 - or in this case, the multi-regional GCS option that this discussion points to. In that case, our attic/borg pricing is 3 cents as compared to (roughly 3 cents) for s3 and 2 cents for GCS - and that assumes you use no bandwidth. Since we charge nothing for usage/bandwidth, the comparable prices from amazon/google would be slightly higher. It sounds like a steal to me and every day plenty of people agree enough to commence using our services. Also, unrelated, you know you can just drive up to an rsync.net location and get your data - even if the Internet is crippled.[1] [1] http://www.rsync.net/products/oob.html http://www.rsync.net/products/oob.html
Corrado 10y ago
My hope is that this will spur AWS to lower their storage prices. Competition is great!
boulos 10y ago
For those that are asking: - Multi-regional is what we used to call Standard (highlighting that it's always been awesome and replicated) - Regional / DRA collapse into one - New offering: Coldline - Massive price cut on operations - Per-object storage class, plus lifecycle (e.g., automatically go from Nearline to Coldline after 60d) Disclosure: I work on Google Cloud (and use Nearline at home!). [Edit: Formatting. I always forget to put two newlines]
Yeroc 10y ago
Google smart to be aggressive on the storage side of their Cloud business. Once you have the data (assuming it's big data) it naturally makes sense to move the processing of that data into Google Cloud services as well...
nodesocket 10y ago
You can attach buckets to cloud instances natively using the gcloud cli[1] as well. Really useful for static file storage. [1] https://cloud.google.com/compute/docs/disks/gcs-buckets https://cloud.google.com/compute/docs/disks/gcs-buckets My only complaint is Google Cloud persistent SSD disk pricing at $0.17 per GB / month. AWS EBS SSD is noticeably lower at $0.12 per GB / month.
stemuk 10y ago
I think there is really no doubt that the storage prices itself ( for all types) are pretty amazing. But let's face it: traffic costs are the huge elephant in the room. By charging 12ct and more per GB, the traffic costs easily become your biggest expense, and make the storage price reduction from 2.6 to 2ct per GB almost forgetable. For me this is in no way acceptable and it seems to be a vicious attempt to sneak in some extra profit without having the customer noticing this upfront. Shure, these harsh words seam like a big exaggeration, but I literally never ever hear or read something about traffic costs in Googles fancy blog posts, and I would make the assumption that only a fraction of the HN community is aware of this fact. 120 bucks for a sloppy TB of traffic is just way too high.
- kyledrake 10y ago
  This x 1000. Even when you throw your own CDN on top of this, the bandwidth markup is nasty. Neocities would be unsustainable if we used GCS for our hosting. I'm paying about $0.01/GB right now, and I've seen market rate at half that. And that's not even directly using IP transit providers. You can get a gigabit unmetered for $450-1000/mo which you can shove a theoretical 324TB through every month. The difference in the numbers is so staggering I sometimes wonder if I'm even doing the math right. Perhaps their bandwidth is better somehow (prove it), but 12-18x better it is probably not, and having truffle shavings added to your IP transit really adds up when you're hauling a lot of traffic. If you're doing something with heavy BW usage and low margins, be careful with stuff like this. It quickly becomes much more expensive than doing it yourself. I'd love to be wrong here. I'm sitting next to 60 pounds of storage servers I'm setting up for a data center, they're taking up my entire living room. I would love to get out of the data persistence business forever. But at these BW rates, it's never going to happen.
  - bedros 10y ago
    out of curiosity, what storage do you use? any experience using backblaze b2 as hot storage, and glacier or google coldline as backup
    - 10y ago
naiv 10y ago
I am confused. Is multi regional the same as availability zones in s3? Or do we upload to the European region and it gets replicated to the US automagically?
ericd 10y ago
I'm extremely reluctant to rely on Google services for core infrastructure for anything revenue generating after seeing the shenanigans they've played on pricing with their other services (especially Google Maps, where they initially jacked up the pricing to an absurd level, then backpedaled and made the prices 1/10 what they were initially threatening) and the relatively short migration windows they typically provide. Besides capricious pricing changes, their support is notoriously bad, even for paid apps for business. They don't seem to understand business customers needs nearly as well as Microsoft/Amazon, from what I've seen. Really not who I want running my business's server infrastructure, even if it's completely free.
banterfoil 10y ago
Can someone explain to my why I shouldn't be worried about Google becoming a monopoly? I think they make some fantastic products and they have been expanding their reach to hardware, DNS, web hosting, and storage.
- Yeroc 10y ago
  You don't need to start worrying about them becoming a monopoly until they actually figure out how to properly provide support for their products to the people (eg enterprises) that need it. :P
throw2016 10y ago
It's difficult to see how AWS and GCE can justify their ridiculous bandwidth charges given how low they buy it in bulk. This is plain profiteering. With cloud bandwidth was supposed to become something one takes for granted not something to fuss over with calculators. And the whole nickel and diming charges that force users to needlessly seperate their computing needs into storage, compute, memory, bandwidth, reads, writes and what not are actually forcing considerable complexity on users. This cannot be brushed aside as a good model for cloud when valuable user time is being wasted grappling with needless intricacies that have no reason not be flat. End users cannot buy bandwidth at low bulk rates and for things like backups may be faced with considerable overcharges on their local connections.

215 points by AndrewDucker 10y ago

Google is making their cloud offering very attractive compared to AWS. The issue with Glacier is the convoluted retrieval pricing. I understand they want to dissuade people from using it as a primary storage, but the potential for a surprise bill is hard to swallow. Interesting how they still offer unlimited storage through their consumer Amazon Drive service.

Gigablah 10y ago
The AWS web console is a pain point. I had a particularly frustrating experience once where attempting to restore a Glacier storage class file in S3 silently failed due to permission issues (I had forgotten to assign permissions to the bucket owner in my automated upload script), but the console told me that the file would be "available in 3-5 hours". I wasted days due to this. Meanwhile other competitors (e.g. Nearline) promise to have your file available within seconds...
- boulos 10y ago
  Yep. Nearline is great. Just so we're clear though: milliseconds ;). There's no latency penalty to get your bytes back, even with Coldline. We do charge a retrieval fee because the pricing is determined by a "promise" that you won't touch the data frequently (roughly once a month for Nearline and once a quarter for Coldline), otherwise you should use our regular flavors. Disclosure: I work on Google Cloud and am a happy GCS customer.
user5994461 10y ago
The issue is not limited to Glacier. The "Infrequent Access" is a nightmare as well. Paying for storage, read and writes per GB and per request, rounded to 128kB blocks. WTF.

user5994461 10y ago

Summary: Google Cloud Storage is now (significantly) cheaper, faster and more available than Amazon S3. Also, it supports multi-region whereas S3 does not. Love the direction Google Cloud is taking <3

Touche 10y ago
The best thing about Google Cloud is the Console. I haven't seen a comparable console for managed services. Think about the UX nightmare that is the AWS Management Console and take that complexity and separate it out neatly and make everything available in a fast search box and you have Google Cloud Console. It's the primary reason I've switched over to Datastore and App Engine.
- RhodesianHunter 10y ago
  Hmm. I actually really like the AWS management console. I find that everything is logically organized and easy to manipulate. That could just be because I've been using it for a long time.
  - rstupek 10y ago
    I agree with you. Google's console is neigh unusable
tehbeard 10y ago
What about ~~reliability~~ durability of storage (bitrot, not the uptime)? I didn't see any information comparing this, is it practically the same across both? edit: durability, not reliability.
- shshhdhs 10y ago
  I think that you mean durability
- klodolph 10y ago
  At this point, durability is hard to compare because the published numbers are total fantasies. S3's published durability is 11 9s, which I've complained about in the past because the chance that the human race will be wiped out next year by an asteroid impact is probably not too far from one in a hundred million, which puts a hard cap on durability at around eight nines. Then ask yourself, "is the chance that someone made a mistake in designing the system which causes catastrophic data loss higher or lower than one in a billion?" Yes, the number is total fantasy. The only actual meaning of the durability SLA is that if the durability SLA is violated, your cloud provider will give you some service credits. If the data is worth more than some service credits, this is small consolation. In practice, if you absolutely must compare durability, what you are trying to do is compare the frequency of black swan events, but actually estimating the frequency of those events requires access to proprietary data which you don't have. Both providers are undoubtedly storing the data in multiple locations on multiple types of storage systems with multiple layers of error coding. After thinking about this issue earlier, the most likely data loss scenario in my opinion was "I get injured, the company hires someone incompetent who doesn't pay the bill for a year while I'm in the hospital."

vbezhenar 10y ago

Coldline seems to be a perfect solution for personal backups. I used amazon glacier, but got hit with extremely huge bill, when I decided to retrieve my backups, so I'm not going to deal with them anymore, their retrieval pricing is absurd. If there's good UI and scripting solutions for manual and automatic backup into google coldline, I'm in.

matt4077 10y ago
You can just user their command line tool, it's got an rsync mode: gsutil -m rsync -P -d -r -x "node_modules|tmp|..."-e ~/ gs://you-will-spend-too-much-time-thinking-of-a-clever-bucket-name (although I should expand it a bit to better handle ignoring stuff by parsing .gitignore)
- jon-wood 10y ago
  > you-will-spend-too-much-time-thinking-of-a-clever-bucket-name FQDNs to the rescue, I very much doubt anyone but me is going to call a bucket backups.ninjagiraffes.co.uk (and if someone does so now, well, good trolling)
  - dekhn 10y ago
    https://cloud.google.com/storage/docs/domain-name-verification https://cloud.google.com/storage/docs/domain-name-verificati... if you own a domain, only people who pass domain ownership verification can make buckets with that domain suffix. Combined with https://cloud.google.com/storage/docs/hosting-static-website https://cloud.google.com/storage/docs/hosting-static-website it means you can easily host static websites by just populating appropriately named buckets with your domain suffix. I think this is a very cool feature.
    - nsheridan 10y ago
      I really wish AWS would introduce domain verification for 'domain-shaped' bucket names.
10y ago

meirelles 10y ago

Google finally is taking a step over the cloud offerings. I was skeptical about them ~1.5 year ago, they where only following the AWS without big innovations.

Heliosmaster 10y ago

Talking about cold storage, i'd like to Pitch C14 of Online.net (the company behind Scaleway): https://www.online.net/en/c14 https://www.online.net/en/c14 0.2ct per Gb.

plandis 10y ago

Multi region support is interesting, but haven't most of GCPs outages affected all of their regions?

hurstdog 10y ago
Checking specifically for GCS at https://status.cloud.google.com/summary https://status.cloud.google.com/summary ... There are two outages there, one that only affected a few projects and one that affected only service in the central US (we have regions over much of the world). Anecdotally, from watching and working with other services internally I don't think most of our outages affect all regions. We actually spend a significant amount of engineering effort ensuring that we're as decoupled as possible. Disclaimer: I'm an Engineering Manager on Google Cloud Storage.
- jonathanoliver 10y ago
  Is there some documentation anywhere talks about how Google Cloud (any product) creates isolation between various regions while at the same time exposing a simple "regionless" programming model?
  - hurstdog 10y ago
    Probably the best reference is the SRE book, specifically the chapters on loadbalancing and distributed consensus protocols. Other than that, the general approach is to minimize global control planes and dependencies in our software stack. In the case of GCS, we do have a single namespace which means we need to look up the locations of data early in the request. Once we know the locations of data we can route the request to the right datacenter to serve it. That global location table is highly replicated and cached, of course. When outages happen, most are caused by changes to the stack, so we also are careful to roll out code or configuration slowly and carefully, slowly increasing the blast radius after it's been proven safe. For example rolling out new binaries first to a few canary instances in one zone, then to a few instances in many regions, then to a full region, then to the world, all spread over a few days. Disclaimer: I'm an Engineering Manager on Google Cloud Storage.

andybak 10y ago

Dear god, the copy on that page is awful. The first paragraph is just vacuous twaddle. Anyone who starts a minor product announcement with the words "Today, we’re excited to announce..." deserves their own circle of hell. Most of that first section is redundant preamble and should be rewritten as bullet points. In fact the whole thing could do with being rewritten in a simple bullet-pointed skimmable form. That way I wouldn't have to come to HN comments to find out what the hell it's all about. Is it too early in the day for a drink?

devopsproject 10y ago
Try some coffee with a light breakfast, then move on to the harder stuff if you still can't cope with "words".
- andybak 10y ago
  A couple of points: 1. It was 4 in the afternoon here when I posted that - a touch too late for breakfast even if it was a bit early for a drink 2. I quite like words when they are strung together well ;-) I still would like to hear someone defend that as a piece of marketing copy. I've re-read it and stand by my earlier diatribe.
  - devopsproject 10y ago
    You've wasted a lot of time on something that is easily ignored. Its time to move on.
    - deleted 10y ago
      [deleted]
    - andybak 10y ago
      And I feel better for it.

brianwawok 10y ago

So what happened to "Durable, reduced availability". It does not seem directly mapped into the new system. I guess it roughly maps to "regional"???

cobookman 10y ago
Yep. Regional is the equivalent of Durable reduced availability.
- brianwawok 10y ago
  Doesn't look like things got auto migrated.. (whereas for the other cases they did get auto migrated)

AdmiralAsshat 10y ago

Cold storage looks cheap at first glance, but for the amount of data I have to backup (about 3 TB), I work out the cost to: (0.7 * / 100) * 3000 = $21 / month * 12 = $252 / yr For that price, I'm better off just paying $60 / yr for CrashPlan, which is unlimited storage. I'd say that applies to anyone who has a terabyte or more of stuff to back up.

devopsproject 10y ago
Have you tried downloading any significant portion of your backup from CrashPlan? Is so, how was performance?
- AdmiralAsshat 10y ago
  Not a significant portion, no. CrashPlan is only my third layer of backup, the offsite backup. Each of my drives have a clone backup drive (typically just a WD My Passport that makes them easy to transport when I travel), and then a larger external that holds backups of all of them--I usually keep that one in storage and only plug it in once a month or so to sync. CP is the third layer of backup. The most I do in a typical week is use the Android app to grab a few movies or documents from the CP storage to get it onto my phone and tablet. I should probably "test" doing a recovery from CrashPlan at some point, but I fear that even getting a third of it (1TB) would probably trip some sort of overcharge from my ISP.
- jrnichols 10y ago
  I have. It was decent. accidentally hosed a vmware image and it was easier for me to just pull the whole thing down from Crashplan. Speeds were decent, limited mostly by my cable modem. I don't recall the exact throughput but I remember looking and saying "Hey, this isn't bad" and not being annoyed by anything. Plus the price isn't too bad.
crudbug 10y ago
https://www.backblaze.com https://www.backblaze.com

kozikow 10y ago

I had to move to AWS entirely due to lack of GPU instances on google cloud. If someone from GCE is reading this - please add GPU instances. Yes, I know unrelated, but I like GCS much more than S3, and I can't use it since by "network" effect caused by GPUs eveyrthing have to run on AWS.

boulos 10y ago
Please ping me with your project id, and your need (e.g., "I want to run TensorFlow on 8 GPUs per VM, 4 VMs at a time, somewhere in Europe"). Disclosure: I work for Google Cloud (contact info in profile).

rsync 10y ago

In case it's not widely known, rsync.net maintains 'gsutil' in the environment (along with s3cmd) so you can move data between google cloud services and rsync.net. ssh user@rsync.net gsutil cp mscdex.exe gs://my-bucket ssh user@rsync.net gsutil rsync -d data gs://mybucket/data Although it should be noted that, circa 2016, the cool kids are all backing up to rsync.net with borg[1][2] (the "holy grail of backup software") which limits the use cases of gsutil with an rsync.net account. HN Readers discount. Just email and ask.[3] [1] https://borgbackup.readthedocs.io/en/stable/ https://borgbackup.readthedocs.io/en/stable/ [2] https://www.stavros.io/posts/holy-grail-backups/ https://www.stavros.io/posts/holy-grail-backups/ [3] info@rsync.net

Veratyr 10y ago
I say this most times I see you advertising and I'll say it again: You really need to work on your pricing to be competitive with things like this. Even with the HN discount your service is expensive and even with the Borg/Attic pricing which is cheaper still, your service is 3x more expensive than alternatives like Nearline (aside from bandwidth), which themselves can be used as a backend for Attic (just mount GCS locally). I really like the look of the service but I can't justify paying over 3x for it.
- rsync 10y ago
  "your service is 3x more expensive than alternatives like Nearline (aside from bandwidth), which themselves can be used as a backend for Attic (just mount GCS locally)." Well, our storage platform is online and random access so it would be inappropriate to compare it to either nearline or glacier. The appropriate comparison is to S3 - or in this case, the multi-regional GCS option that this discussion points to. In that case, our attic/borg pricing is 3 cents as compared to (roughly 3 cents) for s3 and 2 cents for GCS - and that assumes you use no bandwidth. Since we charge nothing for usage/bandwidth, the comparable prices from amazon/google would be slightly higher. It sounds like a steal to me and every day plenty of people agree enough to commence using our services. Also, unrelated, you know you can just drive up to an rsync.net location and get your data - even if the Internet is crippled.[1] [1] http://www.rsync.net/products/oob.html http://www.rsync.net/products/oob.html

Corrado 10y ago

My hope is that this will spur AWS to lower their storage prices. Competition is great!

boulos 10y ago

For those that are asking: - Multi-regional is what we used to call Standard (highlighting that it's always been awesome and replicated) - Regional / DRA collapse into one - New offering: Coldline - Massive price cut on operations - Per-object storage class, plus lifecycle (e.g., automatically go from Nearline to Coldline after 60d) Disclosure: I work on Google Cloud (and use Nearline at home!). [Edit: Formatting. I always forget to put two newlines]

Yeroc 10y ago

Google smart to be aggressive on the storage side of their Cloud business. Once you have the data (assuming it's big data) it naturally makes sense to move the processing of that data into Google Cloud services as well...

nodesocket 10y ago

You can attach buckets to cloud instances natively using the gcloud cli[1] as well. Really useful for static file storage. [1] https://cloud.google.com/compute/docs/disks/gcs-buckets https://cloud.google.com/compute/docs/disks/gcs-buckets My only complaint is Google Cloud persistent SSD disk pricing at $0.17 per GB / month. AWS EBS SSD is noticeably lower at $0.12 per GB / month.

stemuk 10y ago

I think there is really no doubt that the storage prices itself ( for all types) are pretty amazing. But let's face it: traffic costs are the huge elephant in the room. By charging 12ct and more per GB, the traffic costs easily become your biggest expense, and make the storage price reduction from 2.6 to 2ct per GB almost forgetable. For me this is in no way acceptable and it seems to be a vicious attempt to sneak in some extra profit without having the customer noticing this upfront. Shure, these harsh words seam like a big exaggeration, but I literally never ever hear or read something about traffic costs in Googles fancy blog posts, and I would make the assumption that only a fraction of the HN community is aware of this fact. 120 bucks for a sloppy TB of traffic is just way too high.

kyledrake 10y ago
This x 1000. Even when you throw your own CDN on top of this, the bandwidth markup is nasty. Neocities would be unsustainable if we used GCS for our hosting. I'm paying about $0.01/GB right now, and I've seen market rate at half that. And that's not even directly using IP transit providers. You can get a gigabit unmetered for $450-1000/mo which you can shove a theoretical 324TB through every month. The difference in the numbers is so staggering I sometimes wonder if I'm even doing the math right. Perhaps their bandwidth is better somehow (prove it), but 12-18x better it is probably not, and having truffle shavings added to your IP transit really adds up when you're hauling a lot of traffic. If you're doing something with heavy BW usage and low margins, be careful with stuff like this. It quickly becomes much more expensive than doing it yourself. I'd love to be wrong here. I'm sitting next to 60 pounds of storage servers I'm setting up for a data center, they're taking up my entire living room. I would love to get out of the data persistence business forever. But at these BW rates, it's never going to happen.
- bedros 10y ago
  out of curiosity, what storage do you use? any experience using backblaze b2 as hot storage, and glacier or google coldline as backup
  - 10y ago

naiv 10y ago

I am confused. Is multi regional the same as availability zones in s3? Or do we upload to the European region and it gets replicated to the US automagically?

ericd 10y ago

I'm extremely reluctant to rely on Google services for core infrastructure for anything revenue generating after seeing the shenanigans they've played on pricing with their other services (especially Google Maps, where they initially jacked up the pricing to an absurd level, then backpedaled and made the prices 1/10 what they were initially threatening) and the relatively short migration windows they typically provide. Besides capricious pricing changes, their support is notoriously bad, even for paid apps for business. They don't seem to understand business customers needs nearly as well as Microsoft/Amazon, from what I've seen. Really not who I want running my business's server infrastructure, even if it's completely free.

banterfoil 10y ago

Can someone explain to my why I shouldn't be worried about Google becoming a monopoly? I think they make some fantastic products and they have been expanding their reach to hardware, DNS, web hosting, and storage.

Yeroc 10y ago
You don't need to start worrying about them becoming a monopoly until they actually figure out how to properly provide support for their products to the people (eg enterprises) that need it. :P

throw2016 10y ago

It's difficult to see how AWS and GCE can justify their ridiculous bandwidth charges given how low they buy it in bulk. This is plain profiteering. With cloud bandwidth was supposed to become something one takes for granted not something to fuss over with calculators. And the whole nickel and diming charges that force users to needlessly seperate their computing needs into storage, compute, memory, bandwidth, reads, writes and what not are actually forcing considerable complexity on users. This cannot be brushed aside as a good model for cloud when valuable user time is being wasted grappling with needless intricacies that have no reason not be flat. End users cannot buy bandwidth at low bulk rates and for things like backups may be faced with considerable overcharges on their local connections.