3 ms·

>Try running BigQuery on a (big) dataset stored in S3. Seems like a pathological use case to run a query engine on one cloud provider datacenter (Google) again

by jasode 5y ago

>Try running BigQuery on a (big) dataset stored in S3. Seems like a pathological use case to run a query engine on one cloud provider datacenter (Google) against the disk storage at another cloud provider (Amazon). Even if egress were $0, I still wouldn't want to do that. I want queries to run as fast as they can and the WAN link bandwidth is opposed to that. Is there anything about BigQuery that would compel anyone to do that instead of just using AWS RedShift?

paunchy 5y ago
Keep in mind that most of the big three regions are located in the same metro area, often times right across the street from one another. They have private network peering that circumvents WAN circuits, your data is literally transiting between ethernet ports on the same switch. So generally speaking, latency and bandwidth between services is not a significant concern. It's all about egress billing.
zwayhowder 5y ago
My experience in most enterprises is that we don't get to pick all the tools and sometimes we don't even get asked our opinion. Recent case: The data team picked S3 for storage, and they picked Power BI for analysis. Don't ask me why they didn't ask my opinion at the time (What would I know, I'm only the principal cloud architect here). Things like operational overhead don't always get a look in when a team has convinced someone with the purchasing authority that tool X is going to solve all their problems. Even if the entire org has zero experience with it and it's going to have flow on effects. A recent example at one of my customers was a team deciding to outsource a platform to the provider. (Outsource, not SaaS it's a managed service hosted in AWS). I told them the network design and AWS build on our side to join the two would require significant effort and they said that's fine. Now we've spent almost their entire budget for the move on just working out how to connect their VPC to ours (there are some legislated controls we had to put in place and the vendor architects were less than helpfull). Of course it's all my team's fault because we are the ones who say "you can't just plug the two together" and it would be much better if we had a "can-do attitude like the other team instead of naysaying all the time."
dx034 5y ago
It can make a lot of sense. Store data at the provider that can do it best, analyze your data at another provider with a superior product. You'll find data centers of all major providers within a few miles of each other in at least 10 locations around the world. Latencies are <5ms and the links between those data centers are cheap and can provide huge bandwidths (even though they don't want you to know this and still charge huge prices). So from both a technical and an economical perspective there should be no reason why you can't shift terrabytes of data daily between GCS, AWS, Azure, Hetzner, OVH, Cloudflare etc. Only artificially high egress pricing set by the three biggest providers keeps people from doing that. In Europe with OVH and Hetzner that's exactly what a lot of people already do (from my experience), also nicely visualized in their Weathermap (look at Frankfurt) [1]. These two providers are tiny compared to AWS and yet the 400gbit/s links between them are utilized ~50% pretty much 24/7. And from my experience I can say that there are quite a few use cases where mixing these providers is cheap and easy. [1] http://weathermap.ovh.net/#europe http://weathermap.ovh.net/#europe
fragmede 5y ago
RedShift isn't evening even competitive within AWS! Snowflake.com runs on top of AWS and provides a better product than RedShift as a third-party product which means they don't get the same hardware avantages RedShift gets. If you use RedShift to any real degree you owe it to your internal users to talk to one of their salespersons.

by jasode 5y ago

paunchy 5y ago

Keep in mind that most of the big three regions are located in the same metro area, often times right across the street from one another. They have private network peering that circumvents WAN circuits, your data is literally transiting between ethernet ports on the same switch. So generally speaking, latency and bandwidth between services is not a significant concern. It's all about egress billing.

zwayhowder 5y ago

My experience in most enterprises is that we don't get to pick all the tools and sometimes we don't even get asked our opinion. Recent case: The data team picked S3 for storage, and they picked Power BI for analysis. Don't ask me why they didn't ask my opinion at the time (What would I know, I'm only the principal cloud architect here). Things like operational overhead don't always get a look in when a team has convinced someone with the purchasing authority that tool X is going to solve all their problems. Even if the entire org has zero experience with it and it's going to have flow on effects. A recent example at one of my customers was a team deciding to outsource a platform to the provider. (Outsource, not SaaS it's a managed service hosted in AWS). I told them the network design and AWS build on our side to join the two would require significant effort and they said that's fine. Now we've spent almost their entire budget for the move on just working out how to connect their VPC to ours (there are some legislated controls we had to put in place and the vendor architects were less than helpfull). Of course it's all my team's fault because we are the ones who say "you can't just plug the two together" and it would be much better if we had a "can-do attitude like the other team instead of naysaying all the time."

dx034 5y ago

It can make a lot of sense. Store data at the provider that can do it best, analyze your data at another provider with a superior product. You'll find data centers of all major providers within a few miles of each other in at least 10 locations around the world. Latencies are <5ms and the links between those data centers are cheap and can provide huge bandwidths (even though they don't want you to know this and still charge huge prices). So from both a technical and an economical perspective there should be no reason why you can't shift terrabytes of data daily between GCS, AWS, Azure, Hetzner, OVH, Cloudflare etc. Only artificially high egress pricing set by the three biggest providers keeps people from doing that. In Europe with OVH and Hetzner that's exactly what a lot of people already do (from my experience), also nicely visualized in their Weathermap (look at Frankfurt) [1]. These two providers are tiny compared to AWS and yet the 400gbit/s links between them are utilized ~50% pretty much 24/7. And from my experience I can say that there are quite a few use cases where mixing these providers is cheap and easy. [1] http://weathermap.ovh.net/#europe http://weathermap.ovh.net/#europe

fragmede 5y ago

RedShift isn't evening even competitive within AWS! Snowflake.com runs on top of AWS and provides a better product than RedShift as a third-party product which means they don't get the same hardware avantages RedShift gets. If you use RedShift to any real degree you owe it to your internal users to talk to one of their salespersons.