3 ms·

Plenty of Xooglers want "just give me direct access to Colossus". But priority-wise, the market seems to want: 1. "Give me a block device I can boot off of, mo

by boulos 10y ago

Plenty of Xooglers want "just give me direct access to Colossus". But priority-wise, the market seems to want: 1. "Give me a block device I can boot off of, mount, and run anything I want to from a single VM" (PD, which is just built on Colossus) 2. "Give me NFS". 3. "Give me massive I/O" (GCS) I think we're doing fine-ish in 1 and 3. The main competition is a dense pile of drives in a box for Hadoop, but we lean on GCS for that via our HDFS connector (https://cloud.google.com/hadoop/google-cloud-storage-connector https://cloud.google.com/hadoop/google-cloud-storage-connect...). It's our recommended setup, the default for Dataproc, and honestly better in many ways than running in a single Colossus cell (you get failover in case of a zonal outage, and by the same token you can have lots of users simultaneously running Hadoop or other processing jobs in different zones). PS - I'm going to go searching for your whitepaper (I find in arguing with folks that network is the bottleneck for something like a NetApp box, not CPUs). [Edit: Newlines between lists... always forgetting the newlines]

ChuckMcM 10y ago
Of course you the cloud vendor is doing fine, the question was when is it too expensive for your customer. And my thesis is that because storage is not a first class service you can't precisely optimize storage spend (or your storage offering) for you customer and that forces them out of cloud situation into their own managed infrastructure. You could also improve your operational efficiency but that isn't a priority yet at the big G. I expect over time it will become one and you'll figure it out but in the meantime your customer has to over provision the crap out of their resources to meet their performance needs. If Bart is still around it was shared with him and the rest of the 'cost of storage' team back in 2009.

by boulos 10y ago

ChuckMcM 10y ago

Of course you the cloud vendor is doing fine, the question was when is it too expensive for your customer. And my thesis is that because storage is not a first class service you can't precisely optimize storage spend (or your storage offering) for you customer and that forces them out of cloud situation into their own managed infrastructure. You could also improve your operational efficiency but that isn't a priority yet at the big G. I expect over time it will become one and you'll figure it out but in the meantime your customer has to over provision the crap out of their resources to meet their performance needs. If Bart is still around it was shared with him and the rest of the 'cost of storage' team back in 2009.