4 ms·

Drive-by response: I’d ask two questions — how possible is it to isolate the code which is proprietary? …is there a third option which allows not blowing your

Drive-by response: I’d ask two questions — how possible is it to isolate the code which is proprietary? …is there a third option which allows not blowing your budget, but remaining generic? Naively, isolating the proprietary interface in a wrapper in your code minimizes changes should you need to move — and blowing your budget isn’t a long term solution. So does it come down to the proprietary option or fail?

scarface_74 3y ago
This would only come from someone who has never done or managed a migration at scale. There are so many moving pieces doing any migration at scale, your code is the least of the problems. I have seen it take nearly a year to migrate hundreds of generic VMs with VM hosted databases from on prem to cloud. You can’t get more generic than that. Yes, this was with AWS Professional Services (where I use to work) doing much of the work.
- zmgsabst 3y ago
  Yawn, insulting others experience to make your point is trite. Perhaps you could justify your point rather than attack other people and appeal to authority?
  - scarface_74 3y ago
    There is so much involved with any migration at scale - data migration, regression testing, network changes even where a company many have a physical connection to their cloud provider, security, compliance , training, and the list goes on.
    - zmgsabst 3y ago
      Okay — and? How is any of that relevant to what I said? You’re ignoring that maintaining your code without modifying your data access (changing only the wrapper) significantly simplifies virtually everything you listed - with the exception of network topology changes.
      - scarface_74 3y ago
        Because in the grand scheme of things “your code” and even changing your code is just a minuscule part of what it takes if a migration did ever happen. So the question remains, have you ever been a part of a large scale migration? Have you ever been in the room when the planning was happening?
        zmgsabst 3y ago
        No — changing your code makes every one of those aspects harder, more time consuming, and thus a worse option. (Again, except network topology.) And yes — I’ve helped migrate global systems on the peta-scale, for finance where compliance is legally mandated. The idea that changing your code at the same time is irrelevant is laughable: Changing your software significantly while migrating has caused major errors in projects I’ve seen attempt it — you’re essentially doing a rewrite while trying to migrate, which introduces bugs in compliance, security, etc. - - - - Edit to include a specific example: Let’s imagine I’m migrating a dataset, A. And my copy is A’. In the scenario where I have some wrappers, W and W’, I can write my validation test on the data, T, to work against that interface. So then T(W(A)) = T(W’(A’)) implies that W(A) = W’(A’), since T is a function (and the same code, run both places). My problem in demonstrating my data integrity reduces to ensuring that my wrappers faithfully marshal data — something I’ll get by writing unit tests. In the scenario where I don’t have those data wrappers, I have to write two tests — T and T’. But then what does T(A) = T’(A’) prove? Nothing directly — I need to prove that my two integrity tests are actually measuring the same thing. In this way, isolating the complexity of your data marshaling directly impacts the complexity of downstream tasks, such as verifying the integrity of your data.
        scarface_74 3y ago
        Again, you’re looking at your little piece of work that is a speck in an entire organization. Your CTO is working with your CIO, compliance department, finance department, product owners, the PMO organization, etc. They are talking to the board about strategic decisions and what will give them a competitive advantage. He’s not losing sleep over whether you put a facade over your data access layer because in some distant future long after he is gone, AWS may raise prices. If the spend is large enough, he’s going to talk to his dedicated sales rep to lower prices long before he comes to zmgsabt and asks him did he make his data access class “cloud agnostic”. A dependency always slips in somewhere unless you are constantly testing and preparing for portability like Uber does. Hell, I released code that was part of a major official open source “AWS Solution” and got complaints a few weeks later that it was dependent on a region and that’s not the first time I’ve seen that happen. Let alone a dependency on all of the arns always having “aws” in them not thinking they wouldn’t work in China or gov-cloud. I hardcoded the partition. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/pseudo-parameter-reference.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui... Another case is that “we are using MySQL to avoid lock in with AWS”. Years later. “Oh shit. The data team is loading data into the database using the AWS extension that lets you load data from S3 using a sql query”

scarface_74 3y ago

This would only come from someone who has never done or managed a migration at scale. There are so many moving pieces doing any migration at scale, your code is the least of the problems. I have seen it take nearly a year to migrate hundreds of generic VMs with VM hosted databases from on prem to cloud. You can’t get more generic than that. Yes, this was with AWS Professional Services (where I use to work) doing much of the work.

zmgsabst 3y ago
Yawn, insulting others experience to make your point is trite. Perhaps you could justify your point rather than attack other people and appeal to authority?
- scarface_74 3y ago
  There is so much involved with any migration at scale - data migration, regression testing, network changes even where a company many have a physical connection to their cloud provider, security, compliance , training, and the list goes on.
  - zmgsabst 3y ago
    Okay — and? How is any of that relevant to what I said? You’re ignoring that maintaining your code without modifying your data access (changing only the wrapper) significantly simplifies virtually everything you listed - with the exception of network topology changes.
    - scarface_74 3y ago
      Because in the grand scheme of things “your code” and even changing your code is just a minuscule part of what it takes if a migration did ever happen. So the question remains, have you ever been a part of a large scale migration? Have you ever been in the room when the planning was happening?
      - zmgsabst 3y ago
        No — changing your code makes every one of those aspects harder, more time consuming, and thus a worse option. (Again, except network topology.) And yes — I’ve helped migrate global systems on the peta-scale, for finance where compliance is legally mandated. The idea that changing your code at the same time is irrelevant is laughable: Changing your software significantly while migrating has caused major errors in projects I’ve seen attempt it — you’re essentially doing a rewrite while trying to migrate, which introduces bugs in compliance, security, etc. - - - - Edit to include a specific example: Let’s imagine I’m migrating a dataset, A. And my copy is A’. In the scenario where I have some wrappers, W and W’, I can write my validation test on the data, T, to work against that interface. So then T(W(A)) = T(W’(A’)) implies that W(A) = W’(A’), since T is a function (and the same code, run both places). My problem in demonstrating my data integrity reduces to ensuring that my wrappers faithfully marshal data — something I’ll get by writing unit tests. In the scenario where I don’t have those data wrappers, I have to write two tests — T and T’. But then what does T(A) = T’(A’) prove? Nothing directly — I need to prove that my two integrity tests are actually measuring the same thing. In this way, isolating the complexity of your data marshaling directly impacts the complexity of downstream tasks, such as verifying the integrity of your data.
        scarface_74 3y ago
        Again, you’re looking at your little piece of work that is a speck in an entire organization. Your CTO is working with your CIO, compliance department, finance department, product owners, the PMO organization, etc. They are talking to the board about strategic decisions and what will give them a competitive advantage. He’s not losing sleep over whether you put a facade over your data access layer because in some distant future long after he is gone, AWS may raise prices. If the spend is large enough, he’s going to talk to his dedicated sales rep to lower prices long before he comes to zmgsabt and asks him did he make his data access class “cloud agnostic”. A dependency always slips in somewhere unless you are constantly testing and preparing for portability like Uber does. Hell, I released code that was part of a major official open source “AWS Solution” and got complaints a few weeks later that it was dependent on a region and that’s not the first time I’ve seen that happen. Let alone a dependency on all of the arns always having “aws” in them not thinking they wouldn’t work in China or gov-cloud. I hardcoded the partition. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/pseudo-parameter-reference.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui... Another case is that “we are using MySQL to avoid lock in with AWS”. Years later. “Oh shit. The data team is loading data into the database using the AWS extension that lets you load data from S3 using a sql query”