5 ms·

I’ve found this to be true, with one caveat. Most cloud pain people experience is from a misunderstanding / abuse of solutions architecture and could have been

I’ve found this to be true, with one caveat. Most cloud pain people experience is from a misunderstanding / abuse of solutions architecture and could have been avoided with a more thoughtful design. It tends to be a people problem, not a tool problem. However, in my experience cloud vendors sell the snot out of their offerings, and the documentation is closer to marketing than truthful technical documentation. Their products’ genuine performance is a closely guarded proprietary secret, and the only way to find out… e.g. whether Lambdas are fast enough for your use case, or whether AWS RDS cross-region replication is good enough for you… is to run your own performance testing. I’ve been burned enough times by AWS making it difficult to figure out exactly how performant their services are, and I’ve learned to test everything myself for the workloads I’ll be running.

Danjoe4 1y ago
This is exactly why I'd rather get a fat VPS from a reputable provider. As long as the bandwidth is sufficient the only limitation is vertical scaling.
- dlisboa 1y ago
  I'm partial to this, the only thing I've found that is harder to achieve is the "edge" part of cloud services. Having a server at each continent is enough for most needs but having users route to the closest one is not as clear to me. I know about Anycast but not how to make it operational for dynamic web products (not like CDN static assets). Any tips on this?
  - deleted 1y ago
    [deleted]
  - whstl 1y ago
    Someone correct me if I’m wrong but: DIY Anycast is probably beyond most people’s reach, as you need to deal with BGP directly. One cool trick is using GeoDNS to route the same domain to a different IP depending on the location of the user, but there are some caveats of course due to caching and TTL. EDIT: Back to Anycast, there are also some providers who allow you BGP configuration, like those: https://www.virtua.cloud/features/your-ip-space https://www.virtua.cloud/features/your-ip-space - https://us.ovhcloud.com/network/byoip https://us.ovhcloud.com/network/byoip - https://docs.hetzner.com/robot/colocation/pricing/ https://docs.hetzner.com/robot/colocation/pricing/ ... However you still need to get the IPs by yourself, by dealing with your Regional Registry (RIPE in my case, in Europe)
  - stackskipton 1y ago
    You could start using DNS Traffic Shaping where DNS server looks at IP making the request and returns the IP of closest server. Azure/AWS/GCP all have solutions for this and does not require you to use their services. There are probably other DNS providers that can do it as well. Cloudflare can also do this as well but it's probably more expensive than DNS.
  - toast0 1y ago
    To get anycast working, you need BGP, and to get it working well, I think you need a good understanding of BGP and a lot of points of presence and well connected at each. BGP's default metric of distance is number of networks traversed, which does funny things. Say you're in city A where you use transit provider 1 and city B where you use transit provider 2. If a user is in city B and their ISP is only connected to transit provider 1, BGP says deliver your traffic to city A, because then traffic doesn't leave transit provider 1 until it hits your network. So for every transit network you use, you really want to connect to it at all your PoPs, and you probably want to connect to as many transit networks as feasible. If you're already doing multihoming at many sites, it's something to consider; if not, it's probably a whole lot of headache. GeoDNS as others suggested is a good option. Plenty of providers out there, it's not perfect, but it's alright. Less so for web browsers, but you can also direct users to specific servers. Sample performance for each /24 and /48 and send users to the best server based on the statistics, use IP location as a fallback source of info. Etc. Not great for simple websites, more useful for things with interaction and to reduce the time it takes for tcp slow start (and similar) to reach the available bandwidth.
whstl 1y ago
> the documentation is closer to marketing than truthful technical documentation I participated in AWS training and certification given by AWS for a company to obtain a government contract and I can 100% say that the PAID TRAINING itself is also 100% marketing and developer evangelism.
- ivape 1y ago
  Infra will always be full of so much nonsense because it’s really hard to tell successful developers their code and system design is unusable. People use it because they are paid to do so usually, but it’s literally some of the worst product development I’ve ever seen. AWS will hopefully be reduced to natural language soon enough with AI, and their product team can move on (most likely they moved on a long time ago, and the revolving door at the company meant it was going remain a shittily thought out platform in long term maintenance).
  - hylaride 1y ago
    Some things never change. I remember ~20 years ago a bunch of expensive F5s suddenly showing up to our offices because the CTO and enterprise architects were convinced that irules could solve all their performance problems for something that wasn't even cacheable (gaming results) and would have shoved too much of our logic into the underpowered CPUs on them. They were a much nicer, if overpriced, load balancing alternative to the Cisco Content Switch we were using, though.
- ecshafer 1y ago
  100% agree with you. I took a corporate training, and at one point crammed for the developer cert. It it just marketing. There is never a question where the answer is "Just run this service on EC2 yourself". It is about maximizing your usage of AWS services.
  - jrs235 1y ago
    Platform dependency/lockin is never mentioned as a con[cern].
gonzo41 1y ago
I feel like every cloud build meeting should have a moment where everyone has to defend the question "Wait! could this be a regular database with a regular app on a server with a regular cache?"
stego-tech 1y ago
You took the words right out of my mouth. Between aggressive salespeople marketing any given product as a panacea for everything and mandates from above to arbitrarily use X thing to do Y, there’s a lot of just plain bad architecture out there.
osigurdson 1y ago
>> is to run your own performance testing I think they are shooting themselves in the foot with this approach. If you have to run a monte carlo simulation on every one of their services at your own time and expense just to understand performance and costs, people will naturally shy away from such black boxes.
- usui 1y ago
  > people will naturally shy away from such black boxes. I don't this isn't true. In fact, it seems that in the industry, many developers don't proceed with caution and go straight into usage, only to find the problems later down the road. This is a result of intense marketing on the part of cloud providers.
  - ecshafer 1y ago
    The fact is most developers in most companies have very little choice. Many medium to large companies (1k-50k employees) the CTO gets wined and dined by AWS/Azure/Oracle and they decide to move to that cloud. They bring in their solutions architects and do the training. The corporate architects for the divisions set the goals. So the rank and file developers get told that they have to make this work in AWS using RDS and they have almost zero power over this choice.
    - whstl 1y ago
      It doesn't even have to be in companies that big. The AWS salespeople took the CTO and a couple of directors of engineering for diner in a fancy restaurant. That was in a fintech that had around 200 employees. AWS also paid for the mandatory marketing... sorry, mandatory training sessions we tech managers had to do. This is how much it takes for a CTO to demand the next week that "everything should be done with AWS cloud-native stuff if possible".
      - osigurdson 1y ago
        If the wined and dined CTO doesn't care about the costs, then the team shouldn't care either.

Danjoe4 1y ago

This is exactly why I'd rather get a fat VPS from a reputable provider. As long as the bandwidth is sufficient the only limitation is vertical scaling.

dlisboa 1y ago
I'm partial to this, the only thing I've found that is harder to achieve is the "edge" part of cloud services. Having a server at each continent is enough for most needs but having users route to the closest one is not as clear to me. I know about Anycast but not how to make it operational for dynamic web products (not like CDN static assets). Any tips on this?
- deleted 1y ago
  [deleted]
- whstl 1y ago
  Someone correct me if I’m wrong but: DIY Anycast is probably beyond most people’s reach, as you need to deal with BGP directly. One cool trick is using GeoDNS to route the same domain to a different IP depending on the location of the user, but there are some caveats of course due to caching and TTL. EDIT: Back to Anycast, there are also some providers who allow you BGP configuration, like those: https://www.virtua.cloud/features/your-ip-space https://www.virtua.cloud/features/your-ip-space - https://us.ovhcloud.com/network/byoip https://us.ovhcloud.com/network/byoip - https://docs.hetzner.com/robot/colocation/pricing/ https://docs.hetzner.com/robot/colocation/pricing/ ... However you still need to get the IPs by yourself, by dealing with your Regional Registry (RIPE in my case, in Europe)
- stackskipton 1y ago
  You could start using DNS Traffic Shaping where DNS server looks at IP making the request and returns the IP of closest server. Azure/AWS/GCP all have solutions for this and does not require you to use their services. There are probably other DNS providers that can do it as well. Cloudflare can also do this as well but it's probably more expensive than DNS.
- toast0 1y ago
  To get anycast working, you need BGP, and to get it working well, I think you need a good understanding of BGP and a lot of points of presence and well connected at each. BGP's default metric of distance is number of networks traversed, which does funny things. Say you're in city A where you use transit provider 1 and city B where you use transit provider 2. If a user is in city B and their ISP is only connected to transit provider 1, BGP says deliver your traffic to city A, because then traffic doesn't leave transit provider 1 until it hits your network. So for every transit network you use, you really want to connect to it at all your PoPs, and you probably want to connect to as many transit networks as feasible. If you're already doing multihoming at many sites, it's something to consider; if not, it's probably a whole lot of headache. GeoDNS as others suggested is a good option. Plenty of providers out there, it's not perfect, but it's alright. Less so for web browsers, but you can also direct users to specific servers. Sample performance for each /24 and /48 and send users to the best server based on the statistics, use IP location as a fallback source of info. Etc. Not great for simple websites, more useful for things with interaction and to reduce the time it takes for tcp slow start (and similar) to reach the available bandwidth.

whstl 1y ago

> the documentation is closer to marketing than truthful technical documentation I participated in AWS training and certification given by AWS for a company to obtain a government contract and I can 100% say that the PAID TRAINING itself is also 100% marketing and developer evangelism.

ivape 1y ago
Infra will always be full of so much nonsense because it’s really hard to tell successful developers their code and system design is unusable. People use it because they are paid to do so usually, but it’s literally some of the worst product development I’ve ever seen. AWS will hopefully be reduced to natural language soon enough with AI, and their product team can move on (most likely they moved on a long time ago, and the revolving door at the company meant it was going remain a shittily thought out platform in long term maintenance).
- hylaride 1y ago
  Some things never change. I remember ~20 years ago a bunch of expensive F5s suddenly showing up to our offices because the CTO and enterprise architects were convinced that irules could solve all their performance problems for something that wasn't even cacheable (gaming results) and would have shoved too much of our logic into the underpowered CPUs on them. They were a much nicer, if overpriced, load balancing alternative to the Cisco Content Switch we were using, though.
ecshafer 1y ago
100% agree with you. I took a corporate training, and at one point crammed for the developer cert. It it just marketing. There is never a question where the answer is "Just run this service on EC2 yourself". It is about maximizing your usage of AWS services.
- jrs235 1y ago
  Platform dependency/lockin is never mentioned as a con[cern].

gonzo41 1y ago

I feel like every cloud build meeting should have a moment where everyone has to defend the question "Wait! could this be a regular database with a regular app on a server with a regular cache?"

stego-tech 1y ago

You took the words right out of my mouth. Between aggressive salespeople marketing any given product as a panacea for everything and mandates from above to arbitrarily use X thing to do Y, there’s a lot of just plain bad architecture out there.

osigurdson 1y ago

>> is to run your own performance testing I think they are shooting themselves in the foot with this approach. If you have to run a monte carlo simulation on every one of their services at your own time and expense just to understand performance and costs, people will naturally shy away from such black boxes.

usui 1y ago
> people will naturally shy away from such black boxes. I don't this isn't true. In fact, it seems that in the industry, many developers don't proceed with caution and go straight into usage, only to find the problems later down the road. This is a result of intense marketing on the part of cloud providers.
- ecshafer 1y ago
  The fact is most developers in most companies have very little choice. Many medium to large companies (1k-50k employees) the CTO gets wined and dined by AWS/Azure/Oracle and they decide to move to that cloud. They bring in their solutions architects and do the training. The corporate architects for the divisions set the goals. So the rank and file developers get told that they have to make this work in AWS using RDS and they have almost zero power over this choice.
  - whstl 1y ago
    It doesn't even have to be in companies that big. The AWS salespeople took the CTO and a couple of directors of engineering for diner in a fancy restaurant. That was in a fintech that had around 200 employees. AWS also paid for the mandatory marketing... sorry, mandatory training sessions we tech managers had to do. This is how much it takes for a CTO to demand the next week that "everything should be done with AWS cloud-native stuff if possible".
    - osigurdson 1y ago
      If the wined and dined CTO doesn't care about the costs, then the team shouldn't care either.