5 ms·
Good that Google is working to improve in this area. They're light years behind AWS. A couple of weeks ago I contacted their support because our newly created
by Yrlec 11y ago
Good that Google is working to improve in this area. They're light years behind AWS.
A couple of weeks ago I contacted their support because our newly created Google Cloud Transfer Service jobs were failing with "UNKNOWN ERROR". Turned out that it was a permission issue. I pointed out this must be a bug on their part since the page were you create transfer jobs says "Creating a transfer grants a Cloud Storage Transfer Service account the necessary source, destination, and project permissions to complete the transfer." Oddly enough they didn't agree that it was a bug.
Instead gave me a gsutil command to update the permissions. To my big surprise there was no way to set permissions to certain prefixes of the bucket. The command had to loop over ALL items in the bucket and update each permission individually. I pointed out that we have >300 million items in out buckets so this is going to incur large I/O costs (changing permissions is a Class a Request) and take ages to complete. Apparently t there was no other way. The command has now been running for 10 days and it's not even half way through. It has also crashed on several occasions, forcing me to restart it.
- saurik 11y agoDid they at least comp the costs? When I have been in remotely similar situations with Amazon they have said "just do a hundred million API calls and we'll drop the fees, we are sorry this is so hard".
- Yrlec 11y agoNo, they haven't comped anything. It's actually costing us hundreds of $/day because the transfer job is supposed to transfer the items to Nearline, so every day that passes without it working is costing us a lot.
- hurstdog 11y agoIf it helps, I'll dig into this when I get into work. This is the first I've heard of this case. -Andrew, GCS
- hurstdog 11y agoOk, I dug into this a bit. You got stuck in a trap in our current setup. We've improved the docs slightly to make this less common, we're going to fix up the error messages to be more clear, and we have a longer-term plan to make this much more push-button. Lastly, I'm going to go chase people down about getting you a comp for this. Apologies for the trouble. -Andrew
- Yrlec 11y agoThanks a lot! I appreciate it!