8 ms·

People forget AWS is 20 years old. When I routinely remind people of that fact they often remark (incredulously) with "WHAT? Really?" When I think about it the

People forget AWS is 20 years old. When I routinely remind people of that fact they often remark (incredulously) with "WHAT? Really?" When I think about it the entire paradigm of "regions" itself seems completely antiquated in the grand scheme of things. AWS has made a few moves on this but in the end you're largely still tied to this fundamental regional concept in terms of control planes, etc and you incur the ridiculousness that is VPCs, wild billing and availability issues, all kinds of hacks, stacks, and effort all over the place to have multiple regions/AZs, etc. I suspect that later starters like Fly, Cloudflare, etc with their approaches of (more or less) compute at edge with no concept of region etc will grow more and more to dominate the cloud space with AWS region-based approaches primarily relegated to "legacy applications" that are stuck there. It may take a generation or so to get there but more and more the next crop of startups, etc are building on things like Fly instead of AWS and friends. When I think about it the region based approaches are more akin to "hosted elastic datacenter" than they are what could be described as a "true cloud" - where it's just everywhere and not even a consideration.

kortilla 3y ago
The realities of regions don’t go away. It’s a primitive to save you money. If a new entrant doesn’t have it they are either just operating in one region or you’re paying cross-region prices for everything.
- kkielhofner 3y ago
  Cloudflare products and pricing (what I'm most familiar with) are in wild opposition to this view. I've never seen anything even remotely hinting at region or geography in their product line other than geo-routing for load balancing products, headers with geo info for you to do something with, etc. They include the serving "POP" in headers for diagnostic purposes but other than that you have no idea. Where do my Workers run? Don't know, don't care. They are substantially cheaper and offer better availability and response times than their region-based "cloud" competitors. Same for KV, D1, R2, and anything else they come up with as they move further and further in (out?). One would expect with them having the ability to allocate supporting hardware dynamically and globally their cost basis is substantially better than having customer facing and controlled region-based resources that still need to be built out for product support (regardless of usage) AND maintain the excess capacity local to each region to be anywhere near "elastic".
  - grog454 3y ago
    > Where do my Workers run? Don't know, don't care Glad that works for you but for my game servers edge computing is not ideal and generally currently not even applicable.
    - SantaCruz11 3y ago
      Who do you use for game server edge computing? Edgegap?
  - aeyes 3y ago
    Cloudflare also depends on investor money, they haven't made a profit ever and continue to just throw stuff at the wall to grow. At some point in time they'll have to start thinking about profitability.
    - short_sells_poo 3y ago
      It's important to qualify this though. E.g. Amazon was not profitable for a very long time, but only because they furiously reinvested everything. They could've turned profitable at any time. Assuming Cloudflare is a similar case to Amazon: ie they could turn profitable at any time but choose to reinvest those profits immediately into more business, that is a very good spot to be in.
      - applied_heat 3y ago
        Big assumption?
        jamwil 3y ago
        Their financials are public.
        applied_heat 3y ago
        Ah! No need to make any assumptions then.
- 015a 3y ago
  It really doesn't though. Sure, it would if the difference is only "16 mega-regions (regional-model) versus 300 mini-regions (edge)". But that's not the difference. The actual difference, at least how Cloudflare pushes it, is in the software; its a higher layer of abstraction; that customers get to stop thinking about regions and locations. Your software is global; its deployed everywhere; it runs everywhere. This is ENTIRELY antithetical to everything on AWS except CloudFront, Route53, IAM(ish), maybe a couple other minor services. Every Single Thing Every Single AWS Customer configures beyond those global services forces you into a region. Want to deploy a lambda function? Which region would you like? Pricing and available products is different in every region. Fronting it with an API? Where would you like that API Gateway? We'll put CloudFront in front of it so its a bit faster for ya, its global(ish), don't worry. In a no-true-scottsman true-edge world, this goes out the window. Cloudflare Workers is like this; D1 is kinda-ish-there-ish; R2 isn't. Here's the code; spider it across the world; automatically route clients to your closest PoP. No regions. No locations. Its not obvious that AWS will ever make meaningful moves to a world like this, for a ton of reasons, but most meaningfully: it requires customer demand; but AWS's existing customers aren't demanding it, because AWS's customers are old CTOs worried about past quarters too much to think about the next one. The younger, smarter, potential customers who recognize how valuable this is aren't AWS's customers; so AWS never hears from them (which, by the way, should more generally concern Amazon's investors; their "relentless customer driven" approach to their products is great for a few decades, but its absolutely true that most of the time customers just want a faster horse, and you're not hearing from the people who aren't your customers because you've never solved their problems before). So, Amazon has dumped billions upon billions into their couple-dozen giga-regions; you cannot pivot that into hundreds upon hundreds of mini-regions like what Cloudflare (and to a lesser but still awesome degree Fly) has. Vercel is the best case study on this. They are exploding in popularity. They don't run any of their own infra afaik, to be clear, but they whitelabel products from Cloudflare, Neon, and others; all Edge infra providers. No one on Vercel is cross-shopping AWS, and AWS has nothing that feels competitive to what Vercel offers. They can't compete. The outcome, in maybe 10-20 years, of this landscape feels really obvious right now, but AWS's size and the insane scale of their investment into "how we did things in 2010" will turn them into the next IBM. Some would say they already are. There's nothing they can do about it. They're not making the right investments. Even if they wanted to, they can't at the necessary scale, because what they're doing is working right now and their existing customers keep forcing them to spend more money building more structures, more racks, more blades, all in Ashburn VA.
bfung 3y ago
Stateless compute is relatively simple to be region-less, but even fly’s homepage has a map of regions. When state is introduced, then CAP/PACELC distributed systems issues arise, and figuring out approaches to deal with them are a must. Read fly’s Postgres docs, and regions come right back in.
- bob1029 3y ago
  > When state is introduced Only the most trivial business systems can operate without any state. If your application has any sort of shared, mutable state, you absolutely need to pick a region, or go really deep into the EC/consensus/clustering rabbit hole. With a dedicated region with 1 big DB, you can achieve latency figures (over the shared state) that would be infeasible with other schemes. Serializing transactions in a distributed manner is a circus compared to what Postgres has to do to achieve the same. For better or worse, sticking a big SQL database in exactly one region will almost always be the best path for most forms of business. If latency/edge are still a concern at this point, that's when we maybe start talking about more specific geographies and standing up read replicas in those regions. In my experience, if you are peddling B2B webapps, no one will ever complain about east vs west coast latency unless you actually screwed something up in code.
  - swyx 3y ago
    im relatively well read on databases and couldnt figure out - what is the EC in your comment? EC2?
    - bob1029 3y ago
      eventual consistency
  - bfung 3y ago
    Let’s hope someone doesn’t put the backup/replica in different region, the one big instance gonna be tough to restore!
- kkielhofner 3y ago
asim 3y ago
THIS. No one cares about regions, you just care that the app is fast and globally available. The way we did things is a relic of the past and everyone mimics it because trying to guide new user behaviour is hard. The reality is we just need the app to be globally deployed with a globally distributed database and anycast DNS to route you to the local most DC. There's a lot more to it, have built this architecture more than a couple times, but it all needs to be in the service of something and if you're UX isn't better than the next guy, it won't matter much.
- makestuff 3y ago
  Regions do matter for stuff like GDPR though. It is beneficial to know that all of the data is in the EU region and will stay there.
- SamuelAdams 3y ago
  As a counterpoint, anyone working with compliance data needs to know and be certain that data does not cross geo-boundaries. Intelligence data for US gov’t agencies must typically be kept stateside: AWS uses the GovCloud region specifically for satisfying gov compliance requirements. Passing HIPAA data through other countries when not needed to is generally frowned upon - data should be kept as local as reasonably possible. I haven’t worked with GDPR requirements but I have heard they are similar. AWS solved this problem well with regions. I’m curious what other providers that are “regionless” do to solve the compliance problem.
  - kkielhofner 3y ago
    Cloudflare does this although to my knowledge it's not available without an Enterprise subscription which is the only thing they offer with this level of compliance anyway. All lower plans maybe get PCI DSS and that's it. It's a valid counter to my point - while they clearly can do it locking it behind an enterprise "talk to us" subscription is lame. That said they are generally customer, market, and product savvy and I suspect they know that any/most orgs that are able to pull off the broader process, etc for actual compliance with these requirements are in fundamentally different positions than some random startup with a bunch of consumer data. We're also seeing more and more ambiguity with coaching platform startups, etc who don't have a covered entity in sight but that's a different topic for a different day. I've spent most of my career in HIPAA-relevant startups. Generally, I think it's actually a disservice all around for Amazon to basically rubber stamp these architectures and solutions with a HIPAA BAA so that companies think they can call themselves "HIPAA compliant". That's not even what it's supposed to mean, let alone the virtually endless issues and process that need to happen throughout the org for "compliant" to actually mean anything. I haven't reviewed it but I'm certain the AWS BAA is so specific and exclusionary it's trivial to step outside of what it covers. Yes these agreements are a piece of the puzzle but it's completely reckless to throw something together in AWS, present the BAA to a customer, and call yourselves "HIPAA compliant". Savvy customers, investors, etc will call you out but otherwise it's mostly just a matter of time before you discover your standalone AWS BAA is actually just one of the things on a 50 point (or whatever) checklist.
cnity 3y ago
The question of whether "the edge" takes off will come down to whether or not the culture at large will swallow the illusion at its heart. You cannot remove the concept of a region any more than you can remove the concept of "the computer" in a cloud environment.
- kkielhofner 3y ago
  "The cloud is just someone else's computer" - that ship sailed at least a decade ago.
- pests 3y ago
  > any more than you can remove the concept of "the computer" in a cloud environment But that's just not true. Tons of cloud services completely abstract the computer out of the picture. You are paying for capacity or throughput as an abstract unit of cost and the cloud configures as many computers as needed to run your request. You never interact with anything resembling a computer in this situation.
  - lightbendover 3y ago
    Except, you always interact with computerS no matter what provider or architecture you use. It's inescapable as long as you're doing business in software. The trade-offs in architecture and regionality do NOT go away just because a company abstracts over it. They may be able to set up read replicas and consensus on the fly and be able to detect when various approaches make the most sense, but you are still at the mercy of the accompanying constraints. For most use cases, none of this may matter. When you do run into a use case where it matters, you should be fully aware of what trade-offs you're making and have control over them. Not to mention, regions matter when you're serious. GDPR/DSA/DMA/India/China -- the list goes on. Certain data must live in certain bubbles.
washywashy 3y ago
Regions also important for more regulated geographies
alpha_squared 3y ago
> It may take a generation or so to get there but more and more the next crop of startups, etc are building on things like Fly instead of AWS and friends. I'm currently at a startup that started on Aptible a few years ago because there was a lack of expertise in systems development. We're now in the process of migrating to AWS natively because of cost savings (it's why I was hired). Aptible is costing us 3-4x what native AWS will. Yes, we're incurring some operational overhead, but the automation and tooling for managing infra is reasonably mature (Terragrunt & Terraform aren't perfect, but they work well enough). Ultimately, startups may start with higher-level cloud providers, but they'll eventually migrate to lower-level providers eventually. At the higher end of the growth curve, companies likely wind up getting back to on-prem in some way.
scarface_74 3y ago
I work at AWS all opinions are my own. AWS does have local zones and has Lambda at Edge. I will be the first to admit though that the local zones are tied to regional zones and that some global services are tied to us-east-1 like IAM.
yencabulator 3y ago
These days, "edge" just means "has more than N data centers" (where N depends on the speaker). Fly and Cloudflare have regions too, try `flyctl platform regions` or see https://www.cloudflare.com/network/ https://www.cloudflare.com/network/
- jgrahamc 3y ago
  Cloudflare does not have "regions": your stuff runs everywhere on our network. And we can be really smart about where's the best place: https://blog.cloudflare.com/announcing-workers-smart-placement/ https://blog.cloudflare.com/announcing-workers-smart-placeme...
  - yencabulator 3y ago
    That sounds like "Cloudflare automated region handling", not that regions don't exist. For example, make your app talk to two backends in different EC2 regions, and now none of the smart placement helps you. The poster we're replying to seems to be gleefully ignorant of speed of light limits and fundamental distributed systems challenges.
    - jgrahamc 3y ago
      No. That’s wrong. We don’t “manage regions”. 300 data centers. Everything running everything. Only time you’d need to think of a region is if you have some compliance situation (“keep all my data in North America” or “only terminate TLS in the EU”) and ask to subdivide our network using Regional Services. But there’s no decision on here you say “I’ll use Cloudflare-EU-North-42” and get stuck with it as no such things exist.

kortilla 3y ago

The realities of regions don’t go away. It’s a primitive to save you money. If a new entrant doesn’t have it they are either just operating in one region or you’re paying cross-region prices for everything.

kkielhofner 3y ago
Cloudflare products and pricing (what I'm most familiar with) are in wild opposition to this view. I've never seen anything even remotely hinting at region or geography in their product line other than geo-routing for load balancing products, headers with geo info for you to do something with, etc. They include the serving "POP" in headers for diagnostic purposes but other than that you have no idea. Where do my Workers run? Don't know, don't care. They are substantially cheaper and offer better availability and response times than their region-based "cloud" competitors. Same for KV, D1, R2, and anything else they come up with as they move further and further in (out?). One would expect with them having the ability to allocate supporting hardware dynamically and globally their cost basis is substantially better than having customer facing and controlled region-based resources that still need to be built out for product support (regardless of usage) AND maintain the excess capacity local to each region to be anywhere near "elastic".
- grog454 3y ago
  > Where do my Workers run? Don't know, don't care Glad that works for you but for my game servers edge computing is not ideal and generally currently not even applicable.
  - SantaCruz11 3y ago
    Who do you use for game server edge computing? Edgegap?
- aeyes 3y ago
  Cloudflare also depends on investor money, they haven't made a profit ever and continue to just throw stuff at the wall to grow. At some point in time they'll have to start thinking about profitability.
  - short_sells_poo 3y ago
    It's important to qualify this though. E.g. Amazon was not profitable for a very long time, but only because they furiously reinvested everything. They could've turned profitable at any time. Assuming Cloudflare is a similar case to Amazon: ie they could turn profitable at any time but choose to reinvest those profits immediately into more business, that is a very good spot to be in.
    - applied_heat 3y ago
      Big assumption?
      - jamwil 3y ago
        Their financials are public.
        applied_heat 3y ago
        Ah! No need to make any assumptions then.
015a 3y ago
It really doesn't though. Sure, it would if the difference is only "16 mega-regions (regional-model) versus 300 mini-regions (edge)". But that's not the difference. The actual difference, at least how Cloudflare pushes it, is in the software; its a higher layer of abstraction; that customers get to stop thinking about regions and locations. Your software is global; its deployed everywhere; it runs everywhere. This is ENTIRELY antithetical to everything on AWS except CloudFront, Route53, IAM(ish), maybe a couple other minor services. Every Single Thing Every Single AWS Customer configures beyond those global services forces you into a region. Want to deploy a lambda function? Which region would you like? Pricing and available products is different in every region. Fronting it with an API? Where would you like that API Gateway? We'll put CloudFront in front of it so its a bit faster for ya, its global(ish), don't worry. In a no-true-scottsman true-edge world, this goes out the window. Cloudflare Workers is like this; D1 is kinda-ish-there-ish; R2 isn't. Here's the code; spider it across the world; automatically route clients to your closest PoP. No regions. No locations. Its not obvious that AWS will ever make meaningful moves to a world like this, for a ton of reasons, but most meaningfully: it requires customer demand; but AWS's existing customers aren't demanding it, because AWS's customers are old CTOs worried about past quarters too much to think about the next one. The younger, smarter, potential customers who recognize how valuable this is aren't AWS's customers; so AWS never hears from them (which, by the way, should more generally concern Amazon's investors; their "relentless customer driven" approach to their products is great for a few decades, but its absolutely true that most of the time customers just want a faster horse, and you're not hearing from the people who aren't your customers because you've never solved their problems before). So, Amazon has dumped billions upon billions into their couple-dozen giga-regions; you cannot pivot that into hundreds upon hundreds of mini-regions like what Cloudflare (and to a lesser but still awesome degree Fly) has. Vercel is the best case study on this. They are exploding in popularity. They don't run any of their own infra afaik, to be clear, but they whitelabel products from Cloudflare, Neon, and others; all Edge infra providers. No one on Vercel is cross-shopping AWS, and AWS has nothing that feels competitive to what Vercel offers. They can't compete. The outcome, in maybe 10-20 years, of this landscape feels really obvious right now, but AWS's size and the insane scale of their investment into "how we did things in 2010" will turn them into the next IBM. Some would say they already are. There's nothing they can do about it. They're not making the right investments. Even if they wanted to, they can't at the necessary scale, because what they're doing is working right now and their existing customers keep forcing them to spend more money building more structures, more racks, more blades, all in Ashburn VA.

bfung 3y ago

Stateless compute is relatively simple to be region-less, but even fly’s homepage has a map of regions. When state is introduced, then CAP/PACELC distributed systems issues arise, and figuring out approaches to deal with them are a must. Read fly’s Postgres docs, and regions come right back in.

bob1029 3y ago
> When state is introduced Only the most trivial business systems can operate without any state. If your application has any sort of shared, mutable state, you absolutely need to pick a region, or go really deep into the EC/consensus/clustering rabbit hole. With a dedicated region with 1 big DB, you can achieve latency figures (over the shared state) that would be infeasible with other schemes. Serializing transactions in a distributed manner is a circus compared to what Postgres has to do to achieve the same. For better or worse, sticking a big SQL database in exactly one region will almost always be the best path for most forms of business. If latency/edge are still a concern at this point, that's when we maybe start talking about more specific geographies and standing up read replicas in those regions. In my experience, if you are peddling B2B webapps, no one will ever complain about east vs west coast latency unless you actually screwed something up in code.
- swyx 3y ago
  im relatively well read on databases and couldnt figure out - what is the EC in your comment? EC2?
  - bob1029 3y ago
    eventual consistency
- bfung 3y ago
  Let’s hope someone doesn’t put the backup/replica in different region, the one big instance gonna be tough to restore!
kkielhofner 3y ago

asim 3y ago

THIS. No one cares about regions, you just care that the app is fast and globally available. The way we did things is a relic of the past and everyone mimics it because trying to guide new user behaviour is hard. The reality is we just need the app to be globally deployed with a globally distributed database and anycast DNS to route you to the local most DC. There's a lot more to it, have built this architecture more than a couple times, but it all needs to be in the service of something and if you're UX isn't better than the next guy, it won't matter much.

makestuff 3y ago
Regions do matter for stuff like GDPR though. It is beneficial to know that all of the data is in the EU region and will stay there.
SamuelAdams 3y ago
As a counterpoint, anyone working with compliance data needs to know and be certain that data does not cross geo-boundaries. Intelligence data for US gov’t agencies must typically be kept stateside: AWS uses the GovCloud region specifically for satisfying gov compliance requirements. Passing HIPAA data through other countries when not needed to is generally frowned upon - data should be kept as local as reasonably possible. I haven’t worked with GDPR requirements but I have heard they are similar. AWS solved this problem well with regions. I’m curious what other providers that are “regionless” do to solve the compliance problem.
- kkielhofner 3y ago
  Cloudflare does this although to my knowledge it's not available without an Enterprise subscription which is the only thing they offer with this level of compliance anyway. All lower plans maybe get PCI DSS and that's it. It's a valid counter to my point - while they clearly can do it locking it behind an enterprise "talk to us" subscription is lame. That said they are generally customer, market, and product savvy and I suspect they know that any/most orgs that are able to pull off the broader process, etc for actual compliance with these requirements are in fundamentally different positions than some random startup with a bunch of consumer data. We're also seeing more and more ambiguity with coaching platform startups, etc who don't have a covered entity in sight but that's a different topic for a different day. I've spent most of my career in HIPAA-relevant startups. Generally, I think it's actually a disservice all around for Amazon to basically rubber stamp these architectures and solutions with a HIPAA BAA so that companies think they can call themselves "HIPAA compliant". That's not even what it's supposed to mean, let alone the virtually endless issues and process that need to happen throughout the org for "compliant" to actually mean anything. I haven't reviewed it but I'm certain the AWS BAA is so specific and exclusionary it's trivial to step outside of what it covers. Yes these agreements are a piece of the puzzle but it's completely reckless to throw something together in AWS, present the BAA to a customer, and call yourselves "HIPAA compliant". Savvy customers, investors, etc will call you out but otherwise it's mostly just a matter of time before you discover your standalone AWS BAA is actually just one of the things on a 50 point (or whatever) checklist.

cnity 3y ago

The question of whether "the edge" takes off will come down to whether or not the culture at large will swallow the illusion at its heart. You cannot remove the concept of a region any more than you can remove the concept of "the computer" in a cloud environment.

kkielhofner 3y ago
"The cloud is just someone else's computer" - that ship sailed at least a decade ago.
pests 3y ago
> any more than you can remove the concept of "the computer" in a cloud environment But that's just not true. Tons of cloud services completely abstract the computer out of the picture. You are paying for capacity or throughput as an abstract unit of cost and the cloud configures as many computers as needed to run your request. You never interact with anything resembling a computer in this situation.
- lightbendover 3y ago
  Except, you always interact with computerS no matter what provider or architecture you use. It's inescapable as long as you're doing business in software. The trade-offs in architecture and regionality do NOT go away just because a company abstracts over it. They may be able to set up read replicas and consensus on the fly and be able to detect when various approaches make the most sense, but you are still at the mercy of the accompanying constraints. For most use cases, none of this may matter. When you do run into a use case where it matters, you should be fully aware of what trade-offs you're making and have control over them. Not to mention, regions matter when you're serious. GDPR/DSA/DMA/India/China -- the list goes on. Certain data must live in certain bubbles.

washywashy 3y ago

Regions also important for more regulated geographies

alpha_squared 3y ago

> It may take a generation or so to get there but more and more the next crop of startups, etc are building on things like Fly instead of AWS and friends. I'm currently at a startup that started on Aptible a few years ago because there was a lack of expertise in systems development. We're now in the process of migrating to AWS natively because of cost savings (it's why I was hired). Aptible is costing us 3-4x what native AWS will. Yes, we're incurring some operational overhead, but the automation and tooling for managing infra is reasonably mature (Terragrunt & Terraform aren't perfect, but they work well enough). Ultimately, startups may start with higher-level cloud providers, but they'll eventually migrate to lower-level providers eventually. At the higher end of the growth curve, companies likely wind up getting back to on-prem in some way.

scarface_74 3y ago

I work at AWS all opinions are my own. AWS does have local zones and has Lambda at Edge. I will be the first to admit though that the local zones are tied to regional zones and that some global services are tied to us-east-1 like IAM.

yencabulator 3y ago

These days, "edge" just means "has more than N data centers" (where N depends on the speaker). Fly and Cloudflare have regions too, try `flyctl platform regions` or see https://www.cloudflare.com/network/ https://www.cloudflare.com/network/

jgrahamc 3y ago
Cloudflare does not have "regions": your stuff runs everywhere on our network. And we can be really smart about where's the best place: https://blog.cloudflare.com/announcing-workers-smart-placement/ https://blog.cloudflare.com/announcing-workers-smart-placeme...
- yencabulator 3y ago
  That sounds like "Cloudflare automated region handling", not that regions don't exist. For example, make your app talk to two backends in different EC2 regions, and now none of the smart placement helps you. The poster we're replying to seems to be gleefully ignorant of speed of light limits and fundamental distributed systems challenges.
  - jgrahamc 3y ago
    No. That’s wrong. We don’t “manage regions”. 300 data centers. Everything running everything. Only time you’d need to think of a region is if you have some compliance situation (“keep all my data in North America” or “only terminate TLS in the EU”) and ask to subdivide our network using Regional Services. But there’s no decision on here you say “I’ll use Cloudflare-EU-North-42” and get stuck with it as no such things exist.