6 ms·

The tech diagram on AWS is insane. This is what I call vendor lock-in porn I'm wondering how much your monthly bill is?

whalesalad 4y ago
Easy enough to move to other clouds if it was defined in something like terraform or even cloudformation. Could even move out of the cloud to your own bare metal services pretty easily, since the various components are so well defined. AWS makes the most sense though as stated in the docs because a lot of the data being processed is in S3, and reading data from S3 inside of most AWS services is free.
jacurtis 4y ago
I am an Infrastructure Architect (aka "Cloud Architect") so I design cloud systems like this on the daily. The "vendor lock-in" argument always makes me laugh. Its the #1 thing I hear all day long. This diagram is actually pretty simple. It looks worse than it is. All it uses are Lambdas (serverless functions), S3 buckets (object storage), and SNS (broadcast/push queues). There appears to be one traditional server in there with EFS, which is just an elastic file system. All of these systems have equivalents in all the major cloud providers. So if the builder of this wanted to move to GCP or Azure, they are not really locked to AWS. This can all be built in another cloud. Now, could you do it in a day? No. Assuming they are building it with Infrastructure as Code (such as Terraform) then they would need to convert the provider and change resource blocks. But this akin to refactoring in a codebase. Its work, but its not terribly difficult. Then they point it to their new cloud and run `terraform apply`. There is almost no way to entirely remove vendor lock-in. The closest you could come is by designing everything yourself on bare metal servers and renting those from a cloud provider. So instead of using a managed queue system, you run some sort of messaging queue on the server. Then you host files on the server's filesystem, and you run the "lambdas" as applications on the server. But that almost causes more headaches than you save or solve for. I look at Cloud Providers as similar to cell phone providers. I know people who live in fear of being locked into a contract with Verizon or something. But really, what are you going to do? You will always need a cell phone. The only other real choice is AT&T or maybe Sprint/TMobile. How often are you really going to switch and what are you really gaining by doing so? Energy spent worrying about being "locked in" to a cloud vendor is energy wasted. Yeah you can move from AWS to Azure or GCP. But that's about it. What do you gain by switching? Probably almost nothing. They are all pretty comparable at this point in reliability, features, and price (GCP is the slight laggard here, but not by much). If Google calls your company and offers you a huge discount to switch, you could still do it. Aside from that, there's minimal incentive to do so. There are a few weird services that AWS has for example that might be considered "lock-in" services. This would be things like AWS Snowball or AWS Groundstation. These don't have comparable systems on other platforms. In the case of Snowball you probably have so much data on AWS that just transferring data would take months (or even years) which could be considered a form of lock-in. tl;dr - This is a very tame arch diagram. A few lambdas, s3 buckets, and messaging queues, all of which have comparable services on all major clouds. There isn't significant vendor lock-in, this could be rebuilt fairly easily (assuming they used IaC) on any major cloud provider.
- ushakov 4y ago
  Hello > This diagram is actually pretty simple The diagram looks like an ad > All it uses are Lambdas (serverless functions), S3 buckets (object storage), and SNS (broadcast/push queues) Do you actually need all of this or do you use it because Amazon tells you to? I know for instance you cannot use Amazon SES without also using S3 and Lambda > So if the builder of this wanted to move to GCP or Azure, they are not really locked to AWS. This can all be built in another cloud You're saying that I cannot move to other cloud provider without my existing code becoming useless? > Assuming they are building it with Infrastructure as Code (such as Terraform) then they would need to convert the provider and change resource blocks What about the data pipelines and business logic? > There is almost no way to entirely remove vendor lock-in There is: avoiding vendor-specific APIs altogether > Closest you could come is by designing everything yourself on bare metal servers and renting those from a cloud provider I don't have to. There are things like Railway, Fly.io, PlanetScale, Supabase, Upstash, Minio, which can work without locking me in > What do you gain by switching? Freedom > There isn't significant vendor lock-in, this could be rebuilt fairly easily (assuming they used IaC) on any major cloud provider You are contradicting yourself
  - manigandham 4y ago
    > "I know for instance you cannot use Amazon SES without also using S3 and Lambda" You can absolutely use SES without S3 and Lambda. I've used it many times in various projects.
    - ushakov 4y ago
      Not on the receiving side https://docs.aws.amazon.com/ses/latest/dg/receiving-email-concepts.html https://docs.aws.amazon.com/ses/latest/dg/receiving-email-co...
      - manigandham 4y ago
        What do you expect SES to do with your mail after receiving it? S3 and Lambda are optional delivery locations, amongst other choices.
gen220 4y ago
Which part is insane? Put another way, how would you implement this without self-managing pieces like nginx/apache, rabbitmq/kafka, or mongodb on a compute instance? The pieces of managed infra they're depending on are all pretty interchangeable from one cloud provider to another. It's not the cheapest way to solve the problem if you discount the cost of management, upgrades, etc. But if you factor those costs in, it's quite competitive.
- ushakov 4y ago
  I would go to cloud providers, which can offer me managed rabbitmq, kafka or mongodb Certainly, I wouldn't get any vendor-specific services like Lambda or SNS (in this example)
  - aequitas 4y ago
    The Lambda functions will just be running your business logic code. Only specific part Lambda provides is the glue. Which is easy to replace with any other cloud or self hosted alternative.
    - alexander0042 4y ago
      The the end of the day, the real reason I went with Lambda to build this is that when I started, I didn't know anything about spinning up a server, building a storage array, or serving an web request. But what I did know was how to write a Python script to extract weather data from a file. The really cool thing about some of these cloud tools is that they let me ignore all of those (very difficult) problems and just focus on the data
      - jftuga 4y ago
        Where is your lamda_handler defined? I searched your repos but could not find it.
      - aequitas 4y ago
        Exactly what the cloud is best used for imho. You can focus on your core problems and still shift to a different solution later on when that becomes more effective cost wise. But still, when I first started in AWS we had lift and shift'ed most of our platform onto AWS and when it came to adding new features I first looked at Lambda instead of building out on the existing platform. The speed at which I could setup the feature and start measuring and iterating on it was so much better compared to doing it on our existing platform. Even though that was pretty ironed out and fully automated.
manigandham 4y ago
It's rather simple, just messy to look at. It's a timer that triggers a cluster that processes raw data files and uploads the JSON to a server with a reliable file system. Then a Lambda function processes API requests to read that JSON and serve a response. "Lock-in" is just a trade-off like any other engineering decision, same as what programming language or API schema you choose. AWS is massive and reliable, and these services are cheap and widely available so I don't see much of an issue here.
alexander0042 4y ago
This is like the 5th iteration of that darn diagram, and I still can't quite get it right. The other comments are correct, the underlying setup isn't that complicated, and it wouldn't be too tricky to build it for another cloud, since all that's really happening is a Python script reading a NetCDF file. The real perk to building everything using a server-less approach was that everything scaled down to 1 user (me), and then has scaled up beautifully as more people have signed up. The monthly bill isn't great, but has been manageable through donations so far. I'm handling about 10 million API calls/ month, so lots of Lambda invocations, but they're all thankfully very short with tiny outbound data transfer totals, and luckily inbound is free. If the costs ever get out of hand is to throttle the request rate down, but there's still a few optimizations I can do on the AWS side that should help (ARM here I come!).
- QuinnyPig 4y ago
  Want me to take a gander at it? This is a project I’d love to see thrive.
shepherdjerred 4y ago
This is a pretty standard looking diagram. It probably isn't very expensive to run. AWS diagrams are pretty intimidating until you've built a few things with several AWS services. The benefit is that a lot of maintenance work is taken care of for you, and your costs can be low if you don't need a lot of compute.

Easy enough to move to other clouds if it was defined in something like terraform or even cloudformation. Could even move out of the cloud to your own bare metal services pretty easily, since the various components are so well defined. AWS makes the most sense though as stated in the docs because a lot of the data being processed is in S3, and reading data from S3 inside of most AWS services is free.

jacurtis 4y ago

I am an Infrastructure Architect (aka "Cloud Architect") so I design cloud systems like this on the daily. The "vendor lock-in" argument always makes me laugh. Its the #1 thing I hear all day long. This diagram is actually pretty simple. It looks worse than it is. All it uses are Lambdas (serverless functions), S3 buckets (object storage), and SNS (broadcast/push queues). There appears to be one traditional server in there with EFS, which is just an elastic file system. All of these systems have equivalents in all the major cloud providers. So if the builder of this wanted to move to GCP or Azure, they are not really locked to AWS. This can all be built in another cloud. Now, could you do it in a day? No. Assuming they are building it with Infrastructure as Code (such as Terraform) then they would need to convert the provider and change resource blocks. But this akin to refactoring in a codebase. Its work, but its not terribly difficult. Then they point it to their new cloud and run `terraform apply`. There is almost no way to entirely remove vendor lock-in. The closest you could come is by designing everything yourself on bare metal servers and renting those from a cloud provider. So instead of using a managed queue system, you run some sort of messaging queue on the server. Then you host files on the server's filesystem, and you run the "lambdas" as applications on the server. But that almost causes more headaches than you save or solve for. I look at Cloud Providers as similar to cell phone providers. I know people who live in fear of being locked into a contract with Verizon or something. But really, what are you going to do? You will always need a cell phone. The only other real choice is AT&T or maybe Sprint/TMobile. How often are you really going to switch and what are you really gaining by doing so? Energy spent worrying about being "locked in" to a cloud vendor is energy wasted. Yeah you can move from AWS to Azure or GCP. But that's about it. What do you gain by switching? Probably almost nothing. They are all pretty comparable at this point in reliability, features, and price (GCP is the slight laggard here, but not by much). If Google calls your company and offers you a huge discount to switch, you could still do it. Aside from that, there's minimal incentive to do so. There are a few weird services that AWS has for example that might be considered "lock-in" services. This would be things like AWS Snowball or AWS Groundstation. These don't have comparable systems on other platforms. In the case of Snowball you probably have so much data on AWS that just transferring data would take months (or even years) which could be considered a form of lock-in. tl;dr - This is a very tame arch diagram. A few lambdas, s3 buckets, and messaging queues, all of which have comparable services on all major clouds. There isn't significant vendor lock-in, this could be rebuilt fairly easily (assuming they used IaC) on any major cloud provider.

ushakov 4y ago
Hello > This diagram is actually pretty simple The diagram looks like an ad > All it uses are Lambdas (serverless functions), S3 buckets (object storage), and SNS (broadcast/push queues) Do you actually need all of this or do you use it because Amazon tells you to? I know for instance you cannot use Amazon SES without also using S3 and Lambda > So if the builder of this wanted to move to GCP or Azure, they are not really locked to AWS. This can all be built in another cloud You're saying that I cannot move to other cloud provider without my existing code becoming useless? > Assuming they are building it with Infrastructure as Code (such as Terraform) then they would need to convert the provider and change resource blocks What about the data pipelines and business logic? > There is almost no way to entirely remove vendor lock-in There is: avoiding vendor-specific APIs altogether > Closest you could come is by designing everything yourself on bare metal servers and renting those from a cloud provider I don't have to. There are things like Railway, Fly.io, PlanetScale, Supabase, Upstash, Minio, which can work without locking me in > What do you gain by switching? Freedom > There isn't significant vendor lock-in, this could be rebuilt fairly easily (assuming they used IaC) on any major cloud provider You are contradicting yourself
- manigandham 4y ago
  > "I know for instance you cannot use Amazon SES without also using S3 and Lambda" You can absolutely use SES without S3 and Lambda. I've used it many times in various projects.
  - ushakov 4y ago
    Not on the receiving side https://docs.aws.amazon.com/ses/latest/dg/receiving-email-concepts.html https://docs.aws.amazon.com/ses/latest/dg/receiving-email-co...
    - manigandham 4y ago
      What do you expect SES to do with your mail after receiving it? S3 and Lambda are optional delivery locations, amongst other choices.

gen220 4y ago

Which part is insane? Put another way, how would you implement this without self-managing pieces like nginx/apache, rabbitmq/kafka, or mongodb on a compute instance? The pieces of managed infra they're depending on are all pretty interchangeable from one cloud provider to another. It's not the cheapest way to solve the problem if you discount the cost of management, upgrades, etc. But if you factor those costs in, it's quite competitive.

ushakov 4y ago
I would go to cloud providers, which can offer me managed rabbitmq, kafka or mongodb Certainly, I wouldn't get any vendor-specific services like Lambda or SNS (in this example)
- aequitas 4y ago
  The Lambda functions will just be running your business logic code. Only specific part Lambda provides is the glue. Which is easy to replace with any other cloud or self hosted alternative.
  - alexander0042 4y ago
    The the end of the day, the real reason I went with Lambda to build this is that when I started, I didn't know anything about spinning up a server, building a storage array, or serving an web request. But what I did know was how to write a Python script to extract weather data from a file. The really cool thing about some of these cloud tools is that they let me ignore all of those (very difficult) problems and just focus on the data
    - jftuga 4y ago
      Where is your lamda_handler defined? I searched your repos but could not find it.
    - aequitas 4y ago
      Exactly what the cloud is best used for imho. You can focus on your core problems and still shift to a different solution later on when that becomes more effective cost wise. But still, when I first started in AWS we had lift and shift'ed most of our platform onto AWS and when it came to adding new features I first looked at Lambda instead of building out on the existing platform. The speed at which I could setup the feature and start measuring and iterating on it was so much better compared to doing it on our existing platform. Even though that was pretty ironed out and fully automated.

manigandham 4y ago

It's rather simple, just messy to look at. It's a timer that triggers a cluster that processes raw data files and uploads the JSON to a server with a reliable file system. Then a Lambda function processes API requests to read that JSON and serve a response. "Lock-in" is just a trade-off like any other engineering decision, same as what programming language or API schema you choose. AWS is massive and reliable, and these services are cheap and widely available so I don't see much of an issue here.

alexander0042 4y ago

This is like the 5th iteration of that darn diagram, and I still can't quite get it right. The other comments are correct, the underlying setup isn't that complicated, and it wouldn't be too tricky to build it for another cloud, since all that's really happening is a Python script reading a NetCDF file. The real perk to building everything using a server-less approach was that everything scaled down to 1 user (me), and then has scaled up beautifully as more people have signed up. The monthly bill isn't great, but has been manageable through donations so far. I'm handling about 10 million API calls/ month, so lots of Lambda invocations, but they're all thankfully very short with tiny outbound data transfer totals, and luckily inbound is free. If the costs ever get out of hand is to throttle the request rate down, but there's still a few optimizations I can do on the AWS side that should help (ARM here I come!).

QuinnyPig 4y ago
Want me to take a gander at it? This is a project I’d love to see thrive.

shepherdjerred 4y ago

This is a pretty standard looking diagram. It probably isn't very expensive to run. AWS diagrams are pretty intimidating until you've built a few things with several AWS services. The benefit is that a lot of maintenance work is taken care of for you, and your costs can be low if you don't need a lot of compute.