6 ms·

CTO Headaches: Top cloud-to-cloud migration woes (and how to solve them)

66 points by llarsson 6y ago

reacharavindh 6y ago
“ But what I’ve also seen is that if you are using a managed Kubernetes service, your users and permissions handling is perhaps tied not to Kubernetes role-based access control (RBAC) features, but rather, to cloud-specific offerings. Like AWS Identity and Access Management (IAM). Great service, but ties you to the AWS platform. “ - precisely why Kubernetes as a proxy for “if we build on K8s, we can easily move across clouds” from an infrastructure point of view is not true. If you develop your application for K8S, then yeah, by all means good for you. But, don’t say your reason for taking on the complexity of Kubernetes as a platform is the ability to become multi-cloud easily.
- booleanbetrayal 6y ago
  Four days ago, AWS rolled generalized OIDC support for their cluster auth. Presumably, this makes it easier to transition providers since the user store can remain consistent. https://aws.amazon.com/blogs/containers/introducing-oidc-identity-provider-authentication-amazon-eks/ https://aws.amazon.com/blogs/containers/introducing-oidc-ide...
tyingq 6y ago
To me, if you're using AWS, GCP, or Azure, you should be using their "hard to migrate away" services. That's the value of the brand name cloud platforms. Plumbing that's done for you. If you're going to the trouble of being agnostic, then you might as well deploy to a way cheaper bare metal datacenter and enjoy market rate storage, compute and egress. Running cloud agnostic on AWS, GCP, etc, is the worst of both worlds.
- rataata_jr 6y ago
  Thank you. That brought a new perspective to me.
  - motohagiography 6y ago
    Seconded, I thought it was very insightful.
- villasv 6y ago
  I’ve always said this to our CTO. If we were going to use AWS, we should go all in. Once we stopped trying too hard to avoid lock in, we stopped halfassing our architecture.
  - AtlasBarfed 6y ago
    And if AWS hikes rates (or reduces discounts) on you once their algorithmic AI determines your switching cost exceeds the rate hike? "Oh they would NEVER do that" And these gee-whiz heavy-binding services would NEVER steal your data. The strange thing at work is people bend over backwards to use AWS stuff: "I made this virtual network card that I can detach and attach to another VM when we want to keep the same address for a box!" How about we use DNS. AWS services are great for things like their Postgres as a service and other things that fundamentally have standards based access. But the others like big data streams will be a bit more convenient in the beginning and then you're beholden to their supported features and the fact you are handing them visibility to your data.
    - Tehnix 6y ago
      > And if AWS hikes rates (or reduces discounts) on you once their algorithmic AI determines your switching cost exceeds the rate hike? > "Oh they would NEVER do that" Interestingly, what you insinuate is exactly the opposite of actual reality when you look at what has happened over the years: things have gotten CHEAPER. Off the top of my head: - Lambdas are now billed per 1ms instead of per 100ms which will be a significant cost reduction for everyone using Lambda - EKS started at $200 / month for the cluster and is now at ~$72 / month - Each bump in EC2 instance generation is always cheaper for the same or more performance - DynamoDB added on-demand capacity which is at face value more expensive, but allows you to scale to 0 and will usually result in an overall cost reduction - This December they announced GP3 which can scale storage and IOPS independently allowing you to much cheaper scale up your needs for more IOPS without paying for TBs of unused storage and probably many many more.
      - AtlasBarfed 6y ago
        That's AWS just keeping itself from being irrelevant. Every cloud provider is pushing that, and in order to acquire customers and maintain customers, they have to do that. Right now, times are good. Money is good. Top spot. Discounts if you can negotiate it. But eventually this will commodify more tightly (k8s is just a warning shot). When you buy "binding" AWS services, it is the "nobody got fired for picking IBM" with a BIT more market pressure. AWS non-commodity services are the new IBM mainframe.
- llarsson 6y ago
  OP here. I agree with you: if you know your deployment is, and always shall be, on e.g. AWS, you essentially lose out on great integration if you try to be cloud agnostic. However, as in the introduction: "Whether motivated by the need for a multi-cloud strategy, expenditure minimization, legislative or regulatory demands, or simply to get closer to end users, many organizations find themselves migrating from one cloud to another." I work with many customers that migrate away from the big US providers because GDPR and the "Schrems II" ruling makes it legally questionable to continue to use these services for certain data processing. Or ones that have been bitten hard by cloud outages, causing them to fail to deliver service to their customers. Getting a tiny rebate on your cloud bill that month does not begin to cover the lost reputation due to downtime.
- fatnoah 6y ago
  >If you're going to the trouble of being agnostic, then you might as well deploy to a way cheaper bare metal datacenter and enjoy market rate storage, compute and egress. Interesting. I wonder a which scale this becomes true. It's been a long time, so I'm not current on rates, but I've been involved with a couple migrations from datacenter to AWS and AWS was substantially cheaper for the same level of infra in both cases. These were smaller use cases, though.
- kirktrue 6y ago
  What about the case where you're not migrating away, but do need to support multiple clouds for regulatory, failover, customer requirements, etc. reasons? Is there a way to still leverage the first-class features of, say, AWS and GCP without resorting to one of these issues: - death by abstraction of cloud differences - teams incompletely reimplementing features on each service - giving up and going to bare metal ?
  - llarsson 6y ago
    Not really a way to leverage the different services by cloud vendors, because the services really are different. So it is either the abstractions you mention, or going with self-hosted. And the latter cases, you really start to appreciate cloud native tech such as what you find at the CNCF.
- closeparen 6y ago
  S3, SQS, RDS, etc. are some damn nice primitives and also have analogues across all the clouds. The headcount to run their open source equivalents to the same SLAs is not free either.
- rubiquity 6y ago
  Disclosure: I work at AWS, and might surprise you that I disagree. Opinions my own. I see this line get touted a lot but it is almost never accompanied by any depth. It sounds good, but what does it actually mean in practice? Do you have concrete examples of what using the best parts of AWS, GCP, and Azure are? From where I'm sitting, the best parts of the cloud were created 10-15 years ago: On-demand compute (without dealing with the joys of subnetting, DHCP, and configuring Linux interfaces), object and block storage, managed databases/DNS/load balancers. The newer features tend to lack the things you really need in a production ready service: monitoring, debugging, configuration, etc. The right answer to this multi-cloud question is to stop having so many different components in your stack. Be painstakingly deliberate about what you adopt and have someone who can articulate (ideally in writing for posterity) why you're adopting it. Pick a few foundational primitives and master them operationally. One relational database, one web server, one load balancer, one cache, zero message queues ;). (Spoiler alert: this is what all of the cloud vendors themselves do internally.)
  - dastbe 6y ago
    (also work at aws) i think that spoiler alert is a little overgeneralized. first, there’s plenty of advances in those categories in the past 5 or so years (lambda, fargate, qldb, a bunch more) that move the needle significantly on the versions of things that existed 10-15 years ago. we didn’t collectively get everything right by 2012, and i heartily endorse customers using things that take away operational complexity but are less “complete” compared to the lower level counterparts. second, you missed mentioning that circular dependencies is a very real reason for why cloud vendor service teams can’t (not won’t) use services that would otherwise benefit them. this is a problem that is much less apparent to customers of clouds because, to them, the suite of services is provided as a single flat layer of options. i do agree that you want to make sure any service you use is “complete” for you, but that completeness can exist on a spectrum and sometimes getting onboard early and driving their roadmap can benefit you greatly. i also agree that you want to be conservative with how many services you do use, because each one without exception will become a learning experience.
  - tyingq 6y ago
    "It sounds good, but what does it actually mean in practice?" I think what I was saying is pretty straightforward... "If you're going to the trouble of being agnostic" Meaning that if you're trying to build some sort of facade that makes AWS and GCP look the same, you might as well just make a facade that points at something cheaper than AWS/GCP. AWS/GCP markup for things like internet egress, storage, and compute are high, partially because of all that plumbing. Any facade puts you in a lowest common denominator situation where you're paying for lots of stuff then not using it. Or, in short, "cloud agnostic" running on cloud is a weird idea that throws out lots of the value add. Perhaps there are some exceptions for very simple architectures, as you hint.
- didip 6y ago
  I respectfully disagree. There's a sweet spot where companies can use the lowest foundational building blocks, build their own stacks on top, and save resources (comparatively to running your own datacenter). Things like EC2, EKS, S3, and RDS Aurora can be considered foundational but you don't really have to use Athena, CloudWatch, or RedShift (they are super expensive anyway).
- Terretta 6y ago
  Absolutely, and not said enough — but do this in a risk managed way. Hold yourself accountable to really give that thought. Terretta doctrine on this is “Aggressively exploit CSP differentiation, with a credible migration plan you keep in the drawer.” To borrow an AWS metaphor on “One way doors versus two way doors” decisions, do pre-think, but don’t pre-code. Just think as you design the house about whether you might someday want to go through that door another way. It’s inexpensive now to make design decisions that make that easier later. At an enterprise, fully understand the BCP or business continuity plan. You will likely be stunned how long functions are considered OK to be offline for regulator-compliant BCP planning purposes, relative to the SLAs that I.T. gets yelled at about. Your job before committing to vendor lock in is to write down a plausible plan to get out of the lock in that fits (when effort is doubled twice) inside the BCP plan window, and maintain that plan as part of the platform documentation for the capability in question. When you study the economics of re-investment by Azure, Google, and AWS, it’s impossible for an enterprise to achieve the same capability-evolution curve without floating on that rising tide. The hard part is to figure out how to not get capsized by change waves or get swamped if the CSP pulls the plug in your boat. These are part engineering, part business/procurement/legal problems, but much less costly than holding yourself to either a lowest common denominator or fully abstracted principle.
- jjav 6y ago
  Startups tend to get very attractive credits on AWS et.al. This is carefully designed so that by the time the bills start to roll in and become surprisingly huge, it's way too late to try to unroll the dependencies on AWS. There's definitely value in keeping options open by sticking to standard processes on Linux VMs so there is a painless migration path if and when the CFO freaks out.
rsync 6y ago
I don't know if this will penetrate upward to the CTO level, but 'rclone'[1] is an open source, standards based solution to cloud-to-cloud migration. You can use rclone - and all its powers - without installing or running it locally: ssh user@rsync.net rclone s3:/some/bucket gdrive:/blah ... as there is a cloud storage provider with rclone built into the platform.[2] [1] https://rclone.org/ https://rclone.org/ [2] https://www.rsync.net/products/universal.html https://www.rsync.net/products/universal.html
- llarsson 6y ago
  Yeah, that migrates data files from A to B. And that has tremendous value, as your command line shows! But, it does nothing for making, e.g., the message bus service that AWS offers compatible with the one Azure offers, though, so your application will surely know something's up when it doesn't work any more.
- kirktrue 6y ago
  I would hope a chief technical officer with their salt would realize how powerful this setup is. Unfortunately, the lack of a cute name, branding, and marketing engine behind it may cause some to dismiss it as 'it's too simple, it must be missing something.'
  - sidlls 6y ago
    At a certain scale anything purporting to be as simple as a one-liner CLI program is suspect. Larger organizations get messy, it's (almost) never as simple as "clone from S3 bucket X to drive folder Y".

66 points by llarsson 6y ago

“ But what I’ve also seen is that if you are using a managed Kubernetes service, your users and permissions handling is perhaps tied not to Kubernetes role-based access control (RBAC) features, but rather, to cloud-specific offerings. Like AWS Identity and Access Management (IAM). Great service, but ties you to the AWS platform. “ - precisely why Kubernetes as a proxy for “if we build on K8s, we can easily move across clouds” from an infrastructure point of view is not true. If you develop your application for K8S, then yeah, by all means good for you. But, don’t say your reason for taking on the complexity of Kubernetes as a platform is the ability to become multi-cloud easily.

booleanbetrayal 6y ago
Four days ago, AWS rolled generalized OIDC support for their cluster auth. Presumably, this makes it easier to transition providers since the user store can remain consistent. https://aws.amazon.com/blogs/containers/introducing-oidc-identity-provider-authentication-amazon-eks/ https://aws.amazon.com/blogs/containers/introducing-oidc-ide...

tyingq 6y ago

To me, if you're using AWS, GCP, or Azure, you should be using their "hard to migrate away" services. That's the value of the brand name cloud platforms. Plumbing that's done for you. If you're going to the trouble of being agnostic, then you might as well deploy to a way cheaper bare metal datacenter and enjoy market rate storage, compute and egress. Running cloud agnostic on AWS, GCP, etc, is the worst of both worlds.

rataata_jr 6y ago
Thank you. That brought a new perspective to me.
- motohagiography 6y ago
  Seconded, I thought it was very insightful.
villasv 6y ago
I’ve always said this to our CTO. If we were going to use AWS, we should go all in. Once we stopped trying too hard to avoid lock in, we stopped halfassing our architecture.
- AtlasBarfed 6y ago
  And if AWS hikes rates (or reduces discounts) on you once their algorithmic AI determines your switching cost exceeds the rate hike? "Oh they would NEVER do that" And these gee-whiz heavy-binding services would NEVER steal your data. The strange thing at work is people bend over backwards to use AWS stuff: "I made this virtual network card that I can detach and attach to another VM when we want to keep the same address for a box!" How about we use DNS. AWS services are great for things like their Postgres as a service and other things that fundamentally have standards based access. But the others like big data streams will be a bit more convenient in the beginning and then you're beholden to their supported features and the fact you are handing them visibility to your data.
  - Tehnix 6y ago
    > And if AWS hikes rates (or reduces discounts) on you once their algorithmic AI determines your switching cost exceeds the rate hike? > "Oh they would NEVER do that" Interestingly, what you insinuate is exactly the opposite of actual reality when you look at what has happened over the years: things have gotten CHEAPER. Off the top of my head: - Lambdas are now billed per 1ms instead of per 100ms which will be a significant cost reduction for everyone using Lambda - EKS started at $200 / month for the cluster and is now at ~$72 / month - Each bump in EC2 instance generation is always cheaper for the same or more performance - DynamoDB added on-demand capacity which is at face value more expensive, but allows you to scale to 0 and will usually result in an overall cost reduction - This December they announced GP3 which can scale storage and IOPS independently allowing you to much cheaper scale up your needs for more IOPS without paying for TBs of unused storage and probably many many more.
    - AtlasBarfed 6y ago
      That's AWS just keeping itself from being irrelevant. Every cloud provider is pushing that, and in order to acquire customers and maintain customers, they have to do that. Right now, times are good. Money is good. Top spot. Discounts if you can negotiate it. But eventually this will commodify more tightly (k8s is just a warning shot). When you buy "binding" AWS services, it is the "nobody got fired for picking IBM" with a BIT more market pressure. AWS non-commodity services are the new IBM mainframe.
llarsson 6y ago
OP here. I agree with you: if you know your deployment is, and always shall be, on e.g. AWS, you essentially lose out on great integration if you try to be cloud agnostic. However, as in the introduction: "Whether motivated by the need for a multi-cloud strategy, expenditure minimization, legislative or regulatory demands, or simply to get closer to end users, many organizations find themselves migrating from one cloud to another." I work with many customers that migrate away from the big US providers because GDPR and the "Schrems II" ruling makes it legally questionable to continue to use these services for certain data processing. Or ones that have been bitten hard by cloud outages, causing them to fail to deliver service to their customers. Getting a tiny rebate on your cloud bill that month does not begin to cover the lost reputation due to downtime.
fatnoah 6y ago
>If you're going to the trouble of being agnostic, then you might as well deploy to a way cheaper bare metal datacenter and enjoy market rate storage, compute and egress. Interesting. I wonder a which scale this becomes true. It's been a long time, so I'm not current on rates, but I've been involved with a couple migrations from datacenter to AWS and AWS was substantially cheaper for the same level of infra in both cases. These were smaller use cases, though.
kirktrue 6y ago
What about the case where you're not migrating away, but do need to support multiple clouds for regulatory, failover, customer requirements, etc. reasons? Is there a way to still leverage the first-class features of, say, AWS and GCP without resorting to one of these issues: - death by abstraction of cloud differences - teams incompletely reimplementing features on each service - giving up and going to bare metal ?
- llarsson 6y ago
  Not really a way to leverage the different services by cloud vendors, because the services really are different. So it is either the abstractions you mention, or going with self-hosted. And the latter cases, you really start to appreciate cloud native tech such as what you find at the CNCF.
closeparen 6y ago
S3, SQS, RDS, etc. are some damn nice primitives and also have analogues across all the clouds. The headcount to run their open source equivalents to the same SLAs is not free either.
rubiquity 6y ago
Disclosure: I work at AWS, and might surprise you that I disagree. Opinions my own. I see this line get touted a lot but it is almost never accompanied by any depth. It sounds good, but what does it actually mean in practice? Do you have concrete examples of what using the best parts of AWS, GCP, and Azure are? From where I'm sitting, the best parts of the cloud were created 10-15 years ago: On-demand compute (without dealing with the joys of subnetting, DHCP, and configuring Linux interfaces), object and block storage, managed databases/DNS/load balancers. The newer features tend to lack the things you really need in a production ready service: monitoring, debugging, configuration, etc. The right answer to this multi-cloud question is to stop having so many different components in your stack. Be painstakingly deliberate about what you adopt and have someone who can articulate (ideally in writing for posterity) why you're adopting it. Pick a few foundational primitives and master them operationally. One relational database, one web server, one load balancer, one cache, zero message queues ;). (Spoiler alert: this is what all of the cloud vendors themselves do internally.)
- dastbe 6y ago
  (also work at aws) i think that spoiler alert is a little overgeneralized. first, there’s plenty of advances in those categories in the past 5 or so years (lambda, fargate, qldb, a bunch more) that move the needle significantly on the versions of things that existed 10-15 years ago. we didn’t collectively get everything right by 2012, and i heartily endorse customers using things that take away operational complexity but are less “complete” compared to the lower level counterparts. second, you missed mentioning that circular dependencies is a very real reason for why cloud vendor service teams can’t (not won’t) use services that would otherwise benefit them. this is a problem that is much less apparent to customers of clouds because, to them, the suite of services is provided as a single flat layer of options. i do agree that you want to make sure any service you use is “complete” for you, but that completeness can exist on a spectrum and sometimes getting onboard early and driving their roadmap can benefit you greatly. i also agree that you want to be conservative with how many services you do use, because each one without exception will become a learning experience.
- tyingq 6y ago
  "It sounds good, but what does it actually mean in practice?" I think what I was saying is pretty straightforward... "If you're going to the trouble of being agnostic" Meaning that if you're trying to build some sort of facade that makes AWS and GCP look the same, you might as well just make a facade that points at something cheaper than AWS/GCP. AWS/GCP markup for things like internet egress, storage, and compute are high, partially because of all that plumbing. Any facade puts you in a lowest common denominator situation where you're paying for lots of stuff then not using it. Or, in short, "cloud agnostic" running on cloud is a weird idea that throws out lots of the value add. Perhaps there are some exceptions for very simple architectures, as you hint.
didip 6y ago
I respectfully disagree. There's a sweet spot where companies can use the lowest foundational building blocks, build their own stacks on top, and save resources (comparatively to running your own datacenter). Things like EC2, EKS, S3, and RDS Aurora can be considered foundational but you don't really have to use Athena, CloudWatch, or RedShift (they are super expensive anyway).
Terretta 6y ago
Absolutely, and not said enough — but do this in a risk managed way. Hold yourself accountable to really give that thought. Terretta doctrine on this is “Aggressively exploit CSP differentiation, with a credible migration plan you keep in the drawer.” To borrow an AWS metaphor on “One way doors versus two way doors” decisions, do pre-think, but don’t pre-code. Just think as you design the house about whether you might someday want to go through that door another way. It’s inexpensive now to make design decisions that make that easier later. At an enterprise, fully understand the BCP or business continuity plan. You will likely be stunned how long functions are considered OK to be offline for regulator-compliant BCP planning purposes, relative to the SLAs that I.T. gets yelled at about. Your job before committing to vendor lock in is to write down a plausible plan to get out of the lock in that fits (when effort is doubled twice) inside the BCP plan window, and maintain that plan as part of the platform documentation for the capability in question. When you study the economics of re-investment by Azure, Google, and AWS, it’s impossible for an enterprise to achieve the same capability-evolution curve without floating on that rising tide. The hard part is to figure out how to not get capsized by change waves or get swamped if the CSP pulls the plug in your boat. These are part engineering, part business/procurement/legal problems, but much less costly than holding yourself to either a lowest common denominator or fully abstracted principle.
jjav 6y ago
Startups tend to get very attractive credits on AWS et.al. This is carefully designed so that by the time the bills start to roll in and become surprisingly huge, it's way too late to try to unroll the dependencies on AWS. There's definitely value in keeping options open by sticking to standard processes on Linux VMs so there is a painless migration path if and when the CFO freaks out.

rsync 6y ago

I don't know if this will penetrate upward to the CTO level, but 'rclone'[1] is an open source, standards based solution to cloud-to-cloud migration. You can use rclone - and all its powers - without installing or running it locally: ssh user@rsync.net rclone s3:/some/bucket gdrive:/blah ... as there is a cloud storage provider with rclone built into the platform.[2] [1] https://rclone.org/ https://rclone.org/ [2] https://www.rsync.net/products/universal.html https://www.rsync.net/products/universal.html

llarsson 6y ago
Yeah, that migrates data files from A to B. And that has tremendous value, as your command line shows! But, it does nothing for making, e.g., the message bus service that AWS offers compatible with the one Azure offers, though, so your application will surely know something's up when it doesn't work any more.
kirktrue 6y ago
I would hope a chief technical officer with their salt would realize how powerful this setup is. Unfortunately, the lack of a cute name, branding, and marketing engine behind it may cause some to dismiss it as 'it's too simple, it must be missing something.'
- sidlls 6y ago
  At a certain scale anything purporting to be as simple as a one-liner CLI program is suspect. Larger organizations get messy, it's (almost) never as simple as "clone from S3 bucket X to drive folder Y".