6 ms·

You forgot about the egress fees. Try running BigQuery on a (big) dataset stored in S3. You probably wouldn't even think of that because of how stupid that is a

You forgot about the egress fees. Try running BigQuery on a (big) dataset stored in S3. You probably wouldn't even think of that because of how stupid that is at the moment.

jasode 5y ago
>Try running BigQuery on a (big) dataset stored in S3. Seems like a pathological use case to run a query engine on one cloud provider datacenter (Google) against the disk storage at another cloud provider (Amazon). Even if egress were $0, I still wouldn't want to do that. I want queries to run as fast as they can and the WAN link bandwidth is opposed to that. Is there anything about BigQuery that would compel anyone to do that instead of just using AWS RedShift?
- paunchy 5y ago
  Keep in mind that most of the big three regions are located in the same metro area, often times right across the street from one another. They have private network peering that circumvents WAN circuits, your data is literally transiting between ethernet ports on the same switch. So generally speaking, latency and bandwidth between services is not a significant concern. It's all about egress billing.
- zwayhowder 5y ago
  My experience in most enterprises is that we don't get to pick all the tools and sometimes we don't even get asked our opinion. Recent case: The data team picked S3 for storage, and they picked Power BI for analysis. Don't ask me why they didn't ask my opinion at the time (What would I know, I'm only the principal cloud architect here). Things like operational overhead don't always get a look in when a team has convinced someone with the purchasing authority that tool X is going to solve all their problems. Even if the entire org has zero experience with it and it's going to have flow on effects. A recent example at one of my customers was a team deciding to outsource a platform to the provider. (Outsource, not SaaS it's a managed service hosted in AWS). I told them the network design and AWS build on our side to join the two would require significant effort and they said that's fine. Now we've spent almost their entire budget for the move on just working out how to connect their VPC to ours (there are some legislated controls we had to put in place and the vendor architects were less than helpfull). Of course it's all my team's fault because we are the ones who say "you can't just plug the two together" and it would be much better if we had a "can-do attitude like the other team instead of naysaying all the time."
- dx034 5y ago
  It can make a lot of sense. Store data at the provider that can do it best, analyze your data at another provider with a superior product. You'll find data centers of all major providers within a few miles of each other in at least 10 locations around the world. Latencies are <5ms and the links between those data centers are cheap and can provide huge bandwidths (even though they don't want you to know this and still charge huge prices). So from both a technical and an economical perspective there should be no reason why you can't shift terrabytes of data daily between GCS, AWS, Azure, Hetzner, OVH, Cloudflare etc. Only artificially high egress pricing set by the three biggest providers keeps people from doing that. In Europe with OVH and Hetzner that's exactly what a lot of people already do (from my experience), also nicely visualized in their Weathermap (look at Frankfurt) [1]. These two providers are tiny compared to AWS and yet the 400gbit/s links between them are utilized ~50% pretty much 24/7. And from my experience I can say that there are quite a few use cases where mixing these providers is cheap and easy. [1] http://weathermap.ovh.net/#europe http://weathermap.ovh.net/#europe
- fragmede 5y ago
  RedShift isn't evening even competitive within AWS! Snowflake.com runs on top of AWS and provides a better product than RedShift as a third-party product which means they don't get the same hardware avantages RedShift gets. If you use RedShift to any real degree you owe it to your internal users to talk to one of their salespersons.
jerf 5y ago
No, I didn't. This is in the context of a discussion of what happens if AWS lowers egress fees to compete with Cloudflare by going "modular".
GordonS 5y ago
The big 3 have gotten away with crazy egress pricing for too long - I'm really hoping that Cloudflare's R2 puts a huge spotlight on egress bandwidth price gouging by AWS, Azure and GCP, and further hoping that they reduce pricing in response. With the huge margins they must have on egress bandwidth, I'm not holding my breath though.
- datavirtue 5y ago
  Have you ever had to pay for your own business internet connection that is offered by ISPs for the purpose of serving unlimited requests? It is not cheap, at all. Your home and/or business internet connection is a joke compared to that type of service. I don't think they have huge margins on egress at all. There needs to be some incentive for customers of cloud services to minimize bandwidth usage. It is a limited resource.
  - ianlevesque 5y ago
    You don't think they do, but Cloudflare estimated AWS' markup is 7959% here https://blog.cloudflare.com/aws-egregious-egress/ https://blog.cloudflare.com/aws-egregious-egress/
    - vbernat 5y ago
      To move bytes out of network, you need more than a transit contract. You need routers, you need people to operate them. All this is absent from Cloudflare's blog post. With the example provided on South Korea, the conclusion should be that the egress fees are only marginally infuenced by the transit cost.
      - elithrar 5y ago
        The blog post on egress speaks to that - but the true scale at which the major clouds buy hardware and deploy it - just changes the dynamics here. We're (collectively) not used to seeing products marked up to such an extent. The $/Mbps prices there - about $6k/Tbps in the US - are based in reality and are absolutely reflective of what it costs, hardware, software, redundancy and all - for an effectively 1Tbps pipe. If you're pricing as $/GB on top of that capacity and keep it reasonably heavily utilized—which can be hard given diurnal demand—the margins only get better! Products like Glacier (S3) exist to fill exactly those gaps. (Note: currently work at Cloudflare, but wasn't part of this blog and I've been around a bit...)
fairramone 5y ago
This is what BigQuery Omni tries to solve.
moralestapia 5y ago
Heh ... I did something similar without thinking about bandwidth. I had a huge corpus of data and ran some instances that were basically downloading+parsing+summarizing data for a couple days. Again, I didn't think of bandwidth as they were EC2 instances querying S3 objects, what could go wrong, it's Amazon to Amazon, right? Wrong. Bill came up in the 10,000s USD range ... fortunately, it happened during a trial period where I could spend a lot of money for ~three months. I moved my stuff into a small-ish cluster of unmetered VPS and the whole thing is 100x cheaper.

You forgot about the egress fees. Try running BigQuery on a (big) dataset stored in S3. You probably wouldn't even think of that because of how stupid that is at the moment.

jasode 5y ago

>Try running BigQuery on a (big) dataset stored in S3. Seems like a pathological use case to run a query engine on one cloud provider datacenter (Google) against the disk storage at another cloud provider (Amazon). Even if egress were $0, I still wouldn't want to do that. I want queries to run as fast as they can and the WAN link bandwidth is opposed to that. Is there anything about BigQuery that would compel anyone to do that instead of just using AWS RedShift?

paunchy 5y ago
Keep in mind that most of the big three regions are located in the same metro area, often times right across the street from one another. They have private network peering that circumvents WAN circuits, your data is literally transiting between ethernet ports on the same switch. So generally speaking, latency and bandwidth between services is not a significant concern. It's all about egress billing.
zwayhowder 5y ago
My experience in most enterprises is that we don't get to pick all the tools and sometimes we don't even get asked our opinion. Recent case: The data team picked S3 for storage, and they picked Power BI for analysis. Don't ask me why they didn't ask my opinion at the time (What would I know, I'm only the principal cloud architect here). Things like operational overhead don't always get a look in when a team has convinced someone with the purchasing authority that tool X is going to solve all their problems. Even if the entire org has zero experience with it and it's going to have flow on effects. A recent example at one of my customers was a team deciding to outsource a platform to the provider. (Outsource, not SaaS it's a managed service hosted in AWS). I told them the network design and AWS build on our side to join the two would require significant effort and they said that's fine. Now we've spent almost their entire budget for the move on just working out how to connect their VPC to ours (there are some legislated controls we had to put in place and the vendor architects were less than helpfull). Of course it's all my team's fault because we are the ones who say "you can't just plug the two together" and it would be much better if we had a "can-do attitude like the other team instead of naysaying all the time."
dx034 5y ago
It can make a lot of sense. Store data at the provider that can do it best, analyze your data at another provider with a superior product. You'll find data centers of all major providers within a few miles of each other in at least 10 locations around the world. Latencies are <5ms and the links between those data centers are cheap and can provide huge bandwidths (even though they don't want you to know this and still charge huge prices). So from both a technical and an economical perspective there should be no reason why you can't shift terrabytes of data daily between GCS, AWS, Azure, Hetzner, OVH, Cloudflare etc. Only artificially high egress pricing set by the three biggest providers keeps people from doing that. In Europe with OVH and Hetzner that's exactly what a lot of people already do (from my experience), also nicely visualized in their Weathermap (look at Frankfurt) [1]. These two providers are tiny compared to AWS and yet the 400gbit/s links between them are utilized ~50% pretty much 24/7. And from my experience I can say that there are quite a few use cases where mixing these providers is cheap and easy. [1] http://weathermap.ovh.net/#europe http://weathermap.ovh.net/#europe
fragmede 5y ago
RedShift isn't evening even competitive within AWS! Snowflake.com runs on top of AWS and provides a better product than RedShift as a third-party product which means they don't get the same hardware avantages RedShift gets. If you use RedShift to any real degree you owe it to your internal users to talk to one of their salespersons.

jerf 5y ago

No, I didn't. This is in the context of a discussion of what happens if AWS lowers egress fees to compete with Cloudflare by going "modular".

GordonS 5y ago

The big 3 have gotten away with crazy egress pricing for too long - I'm really hoping that Cloudflare's R2 puts a huge spotlight on egress bandwidth price gouging by AWS, Azure and GCP, and further hoping that they reduce pricing in response. With the huge margins they must have on egress bandwidth, I'm not holding my breath though.

datavirtue 5y ago
Have you ever had to pay for your own business internet connection that is offered by ISPs for the purpose of serving unlimited requests? It is not cheap, at all. Your home and/or business internet connection is a joke compared to that type of service. I don't think they have huge margins on egress at all. There needs to be some incentive for customers of cloud services to minimize bandwidth usage. It is a limited resource.
- ianlevesque 5y ago
  You don't think they do, but Cloudflare estimated AWS' markup is 7959% here https://blog.cloudflare.com/aws-egregious-egress/ https://blog.cloudflare.com/aws-egregious-egress/
  - vbernat 5y ago
    To move bytes out of network, you need more than a transit contract. You need routers, you need people to operate them. All this is absent from Cloudflare's blog post. With the example provided on South Korea, the conclusion should be that the egress fees are only marginally infuenced by the transit cost.
    - elithrar 5y ago
      The blog post on egress speaks to that - but the true scale at which the major clouds buy hardware and deploy it - just changes the dynamics here. We're (collectively) not used to seeing products marked up to such an extent. The $/Mbps prices there - about $6k/Tbps in the US - are based in reality and are absolutely reflective of what it costs, hardware, software, redundancy and all - for an effectively 1Tbps pipe. If you're pricing as $/GB on top of that capacity and keep it reasonably heavily utilized—which can be hard given diurnal demand—the margins only get better! Products like Glacier (S3) exist to fill exactly those gaps. (Note: currently work at Cloudflare, but wasn't part of this blog and I've been around a bit...)

fairramone 5y ago

This is what BigQuery Omni tries to solve.

moralestapia 5y ago

Heh ... I did something similar without thinking about bandwidth. I had a huge corpus of data and ran some instances that were basically downloading+parsing+summarizing data for a couple days. Again, I didn't think of bandwidth as they were EC2 instances querying S3 objects, what could go wrong, it's Amazon to Amazon, right? Wrong. Bill came up in the 10,000s USD range ... fortunately, it happened during a trial period where I could spend a lot of money for ~three months. I moved my stuff into a small-ish cluster of unmetered VPS and the whole thing is 100x cheaper.