6 ms·

"More importantly, AWS itself is locked-in to its integrated approach: the entire service is architected both technically and economically to be an all-encompas

by jerf 5y ago

"More importantly, AWS itself is locked-in to its integrated approach: the entire service is architected both technically and economically to be an all-encompassing offering; to modularize itself in response to Cloudflare would be suicidal." Eh, somewhat. AWS is already modular in a lot of ways. You want S3? You got it, no matter where you are. (We're talking after them doing some sort of fee drop here.) You want to run exactly one EC2 instance? No problem. You want a message queue? You don't need anything else. You can integrate it with the notification service but it's optional. Sure, some of their services are integrated, but a lot of that integration is just "this service pulls from S3 and writes to S3", not massive integration at every level. There is some stuff that is deeply tied in, yeah. But it's not like every single AWS service is deeply tied into half the other ones and the moment you open an EC2 instance you also are buying into a dozen other services. (It may feel like it if you put together a network and override the default block storage, but that's really just giving you knobs that are simply preset elsewhere, not really "lockin".) A lot of it is already pretty modular.

H8crilA 5y ago
You forgot about the egress fees. Try running BigQuery on a (big) dataset stored in S3. You probably wouldn't even think of that because of how stupid that is at the moment.
- jasode 5y ago
  >Try running BigQuery on a (big) dataset stored in S3. Seems like a pathological use case to run a query engine on one cloud provider datacenter (Google) against the disk storage at another cloud provider (Amazon). Even if egress were $0, I still wouldn't want to do that. I want queries to run as fast as they can and the WAN link bandwidth is opposed to that. Is there anything about BigQuery that would compel anyone to do that instead of just using AWS RedShift?
  - paunchy 5y ago
    Keep in mind that most of the big three regions are located in the same metro area, often times right across the street from one another. They have private network peering that circumvents WAN circuits, your data is literally transiting between ethernet ports on the same switch. So generally speaking, latency and bandwidth between services is not a significant concern. It's all about egress billing.
  - zwayhowder 5y ago
    My experience in most enterprises is that we don't get to pick all the tools and sometimes we don't even get asked our opinion. Recent case: The data team picked S3 for storage, and they picked Power BI for analysis. Don't ask me why they didn't ask my opinion at the time (What would I know, I'm only the principal cloud architect here). Things like operational overhead don't always get a look in when a team has convinced someone with the purchasing authority that tool X is going to solve all their problems. Even if the entire org has zero experience with it and it's going to have flow on effects. A recent example at one of my customers was a team deciding to outsource a platform to the provider. (Outsource, not SaaS it's a managed service hosted in AWS). I told them the network design and AWS build on our side to join the two would require significant effort and they said that's fine. Now we've spent almost their entire budget for the move on just working out how to connect their VPC to ours (there are some legislated controls we had to put in place and the vendor architects were less than helpfull). Of course it's all my team's fault because we are the ones who say "you can't just plug the two together" and it would be much better if we had a "can-do attitude like the other team instead of naysaying all the time."
  - dx034 5y ago
    It can make a lot of sense. Store data at the provider that can do it best, analyze your data at another provider with a superior product. You'll find data centers of all major providers within a few miles of each other in at least 10 locations around the world. Latencies are <5ms and the links between those data centers are cheap and can provide huge bandwidths (even though they don't want you to know this and still charge huge prices). So from both a technical and an economical perspective there should be no reason why you can't shift terrabytes of data daily between GCS, AWS, Azure, Hetzner, OVH, Cloudflare etc. Only artificially high egress pricing set by the three biggest providers keeps people from doing that. In Europe with OVH and Hetzner that's exactly what a lot of people already do (from my experience), also nicely visualized in their Weathermap (look at Frankfurt) [1]. These two providers are tiny compared to AWS and yet the 400gbit/s links between them are utilized ~50% pretty much 24/7. And from my experience I can say that there are quite a few use cases where mixing these providers is cheap and easy. [1] http://weathermap.ovh.net/#europe http://weathermap.ovh.net/#europe
  - fragmede 5y ago
    RedShift isn't evening even competitive within AWS! Snowflake.com runs on top of AWS and provides a better product than RedShift as a third-party product which means they don't get the same hardware avantages RedShift gets. If you use RedShift to any real degree you owe it to your internal users to talk to one of their salespersons.
- jerf 5y ago
  No, I didn't. This is in the context of a discussion of what happens if AWS lowers egress fees to compete with Cloudflare by going "modular".
- GordonS 5y ago
  The big 3 have gotten away with crazy egress pricing for too long - I'm really hoping that Cloudflare's R2 puts a huge spotlight on egress bandwidth price gouging by AWS, Azure and GCP, and further hoping that they reduce pricing in response. With the huge margins they must have on egress bandwidth, I'm not holding my breath though.
  - datavirtue 5y ago
    Have you ever had to pay for your own business internet connection that is offered by ISPs for the purpose of serving unlimited requests? It is not cheap, at all. Your home and/or business internet connection is a joke compared to that type of service. I don't think they have huge margins on egress at all. There needs to be some incentive for customers of cloud services to minimize bandwidth usage. It is a limited resource.
    - ianlevesque 5y ago
      You don't think they do, but Cloudflare estimated AWS' markup is 7959% here https://blog.cloudflare.com/aws-egregious-egress/ https://blog.cloudflare.com/aws-egregious-egress/
      - vbernat 5y ago
        To move bytes out of network, you need more than a transit contract. You need routers, you need people to operate them. All this is absent from Cloudflare's blog post. With the example provided on South Korea, the conclusion should be that the egress fees are only marginally infuenced by the transit cost.
        elithrar 5y ago
        The blog post on egress speaks to that - but the true scale at which the major clouds buy hardware and deploy it - just changes the dynamics here. We're (collectively) not used to seeing products marked up to such an extent. The $/Mbps prices there - about $6k/Tbps in the US - are based in reality and are absolutely reflective of what it costs, hardware, software, redundancy and all - for an effectively 1Tbps pipe. If you're pricing as $/GB on top of that capacity and keep it reasonably heavily utilized—which can be hard given diurnal demand—the margins only get better! Products like Glacier (S3) exist to fill exactly those gaps. (Note: currently work at Cloudflare, but wasn't part of this blog and I've been around a bit...)
- fairramone 5y ago
  This is what BigQuery Omni tries to solve.
- moralestapia 5y ago
  Heh ... I did something similar without thinking about bandwidth. I had a huge corpus of data and ran some instances that were basically downloading+parsing+summarizing data for a couple days. Again, I didn't think of bandwidth as they were EC2 instances querying S3 objects, what could go wrong, it's Amazon to Amazon, right? Wrong. Bill came up in the 10,000s USD range ... fortunately, it happened during a trial period where I could spend a lot of money for ~three months. I moved my stuff into a small-ish cluster of unmetered VPS and the whole thing is 100x cheaper.
heisenbit 5y ago
Security is a significant integration effort.
zomiaen 5y ago
Not on their backend. S3 goes down, nearly everything else does. We found out last year if Kinesis has issues, so does a bunch of other internal AWS services.
bredren 5y ago
AWS Polly is an example of a service that is tied into others. To do anything more than a demo, Polly requires S3. To get a notification of a completed TTS synthesis, you must use AWS SNS. To get logs you have to use their logging service. I suspect the primitives, like these required ancillary services, are the ones not tied in to each other. But how could they? (AWS probably has found ways) For my project using Polly, I didn’t care. It was kind of interesting to explore AWS some. But I’d guess a lot of folks might not want cloud service primitives. They want cloud products. If they must use primitives to enjoy cloud products, they should want flexibility at least to not be stuck with pricing and feature set on primitives if they don’t compete.
efangs 5y ago
I was with the article until this final point about integrated solutions. I see the argument around data lock-in, but integration is typically non-trivial unless it's coming from the same provider. Plus, I'm certain cloudflare would equally love to provide more services in an integrated way.
Jonovono 5y ago
I think you are missing the point. AWS is modular WITHIN AWS. It's not a decentralised modular system. ie, it can't play well with existing companies because of the hefty egress prices they charge. The point in the article is that, maybe, if you take away the egress fees that opens up a new world where services of different companies can play together nicely (instead of waiting for AWS to implement something), and that creates a new form of innovation that we can't quite predict that could possibly compete with AWS
sudhirj 5y ago
They don’t explain this, but the naming of AWS services indicates how modular they are. Services prefixed with “Amazon” are supposed to usable independently, while services with the “AWS” prefix need to be used inside the ecosystem.

by jerf 5y ago

H8crilA 5y ago

You forgot about the egress fees. Try running BigQuery on a (big) dataset stored in S3. You probably wouldn't even think of that because of how stupid that is at the moment.

jasode 5y ago
>Try running BigQuery on a (big) dataset stored in S3. Seems like a pathological use case to run a query engine on one cloud provider datacenter (Google) against the disk storage at another cloud provider (Amazon). Even if egress were $0, I still wouldn't want to do that. I want queries to run as fast as they can and the WAN link bandwidth is opposed to that. Is there anything about BigQuery that would compel anyone to do that instead of just using AWS RedShift?
- paunchy 5y ago
  Keep in mind that most of the big three regions are located in the same metro area, often times right across the street from one another. They have private network peering that circumvents WAN circuits, your data is literally transiting between ethernet ports on the same switch. So generally speaking, latency and bandwidth between services is not a significant concern. It's all about egress billing.
- zwayhowder 5y ago
  My experience in most enterprises is that we don't get to pick all the tools and sometimes we don't even get asked our opinion. Recent case: The data team picked S3 for storage, and they picked Power BI for analysis. Don't ask me why they didn't ask my opinion at the time (What would I know, I'm only the principal cloud architect here). Things like operational overhead don't always get a look in when a team has convinced someone with the purchasing authority that tool X is going to solve all their problems. Even if the entire org has zero experience with it and it's going to have flow on effects. A recent example at one of my customers was a team deciding to outsource a platform to the provider. (Outsource, not SaaS it's a managed service hosted in AWS). I told them the network design and AWS build on our side to join the two would require significant effort and they said that's fine. Now we've spent almost their entire budget for the move on just working out how to connect their VPC to ours (there are some legislated controls we had to put in place and the vendor architects were less than helpfull). Of course it's all my team's fault because we are the ones who say "you can't just plug the two together" and it would be much better if we had a "can-do attitude like the other team instead of naysaying all the time."
- dx034 5y ago
  It can make a lot of sense. Store data at the provider that can do it best, analyze your data at another provider with a superior product. You'll find data centers of all major providers within a few miles of each other in at least 10 locations around the world. Latencies are <5ms and the links between those data centers are cheap and can provide huge bandwidths (even though they don't want you to know this and still charge huge prices). So from both a technical and an economical perspective there should be no reason why you can't shift terrabytes of data daily between GCS, AWS, Azure, Hetzner, OVH, Cloudflare etc. Only artificially high egress pricing set by the three biggest providers keeps people from doing that. In Europe with OVH and Hetzner that's exactly what a lot of people already do (from my experience), also nicely visualized in their Weathermap (look at Frankfurt) [1]. These two providers are tiny compared to AWS and yet the 400gbit/s links between them are utilized ~50% pretty much 24/7. And from my experience I can say that there are quite a few use cases where mixing these providers is cheap and easy. [1] http://weathermap.ovh.net/#europe http://weathermap.ovh.net/#europe
- fragmede 5y ago
  RedShift isn't evening even competitive within AWS! Snowflake.com runs on top of AWS and provides a better product than RedShift as a third-party product which means they don't get the same hardware avantages RedShift gets. If you use RedShift to any real degree you owe it to your internal users to talk to one of their salespersons.
jerf 5y ago
No, I didn't. This is in the context of a discussion of what happens if AWS lowers egress fees to compete with Cloudflare by going "modular".
GordonS 5y ago
The big 3 have gotten away with crazy egress pricing for too long - I'm really hoping that Cloudflare's R2 puts a huge spotlight on egress bandwidth price gouging by AWS, Azure and GCP, and further hoping that they reduce pricing in response. With the huge margins they must have on egress bandwidth, I'm not holding my breath though.
- datavirtue 5y ago
  Have you ever had to pay for your own business internet connection that is offered by ISPs for the purpose of serving unlimited requests? It is not cheap, at all. Your home and/or business internet connection is a joke compared to that type of service. I don't think they have huge margins on egress at all. There needs to be some incentive for customers of cloud services to minimize bandwidth usage. It is a limited resource.
  - ianlevesque 5y ago
    You don't think they do, but Cloudflare estimated AWS' markup is 7959% here https://blog.cloudflare.com/aws-egregious-egress/ https://blog.cloudflare.com/aws-egregious-egress/
    - vbernat 5y ago
      To move bytes out of network, you need more than a transit contract. You need routers, you need people to operate them. All this is absent from Cloudflare's blog post. With the example provided on South Korea, the conclusion should be that the egress fees are only marginally infuenced by the transit cost.
      - elithrar 5y ago
        The blog post on egress speaks to that - but the true scale at which the major clouds buy hardware and deploy it - just changes the dynamics here. We're (collectively) not used to seeing products marked up to such an extent. The $/Mbps prices there - about $6k/Tbps in the US - are based in reality and are absolutely reflective of what it costs, hardware, software, redundancy and all - for an effectively 1Tbps pipe. If you're pricing as $/GB on top of that capacity and keep it reasonably heavily utilized—which can be hard given diurnal demand—the margins only get better! Products like Glacier (S3) exist to fill exactly those gaps. (Note: currently work at Cloudflare, but wasn't part of this blog and I've been around a bit...)
fairramone 5y ago
This is what BigQuery Omni tries to solve.
moralestapia 5y ago
Heh ... I did something similar without thinking about bandwidth. I had a huge corpus of data and ran some instances that were basically downloading+parsing+summarizing data for a couple days. Again, I didn't think of bandwidth as they were EC2 instances querying S3 objects, what could go wrong, it's Amazon to Amazon, right? Wrong. Bill came up in the 10,000s USD range ... fortunately, it happened during a trial period where I could spend a lot of money for ~three months. I moved my stuff into a small-ish cluster of unmetered VPS and the whole thing is 100x cheaper.

heisenbit 5y ago

Security is a significant integration effort.

zomiaen 5y ago

Not on their backend. S3 goes down, nearly everything else does. We found out last year if Kinesis has issues, so does a bunch of other internal AWS services.

bredren 5y ago

AWS Polly is an example of a service that is tied into others. To do anything more than a demo, Polly requires S3. To get a notification of a completed TTS synthesis, you must use AWS SNS. To get logs you have to use their logging service. I suspect the primitives, like these required ancillary services, are the ones not tied in to each other. But how could they? (AWS probably has found ways) For my project using Polly, I didn’t care. It was kind of interesting to explore AWS some. But I’d guess a lot of folks might not want cloud service primitives. They want cloud products. If they must use primitives to enjoy cloud products, they should want flexibility at least to not be stuck with pricing and feature set on primitives if they don’t compete.

efangs 5y ago

I was with the article until this final point about integrated solutions. I see the argument around data lock-in, but integration is typically non-trivial unless it's coming from the same provider. Plus, I'm certain cloudflare would equally love to provide more services in an integrated way.

Jonovono 5y ago

I think you are missing the point. AWS is modular WITHIN AWS. It's not a decentralised modular system. ie, it can't play well with existing companies because of the hefty egress prices they charge. The point in the article is that, maybe, if you take away the egress fees that opens up a new world where services of different companies can play together nicely (instead of waiting for AWS to implement something), and that creates a new form of innovation that we can't quite predict that could possibly compete with AWS

sudhirj 5y ago

They don’t explain this, but the naming of AWS services indicates how modular they are. Services prefixed with “Amazon” are supposed to usable independently, while services with the “AWS” prefix need to be used inside the ecosystem.