4 ms·

how do y'all look at systems like AWS Aurora, GCP Spanner, etc which offer big speedups and cost savings, but lock you into the platform in practice? this isn'

by asah 3y ago

how do y'all look at systems like AWS Aurora, GCP Spanner, etc which offer big speedups and cost savings, but lock you into the platform in practice? this isn't academic for me - I've got a service that's on vanilla RDS PostgreSQL but growing fast, and I'll soon need to decide whether to move to something cheaper/faster but proprietary or blow my budget trying to stay generic...

zmgsabst 3y ago
Drive-by response: I’d ask two questions — how possible is it to isolate the code which is proprietary? …is there a third option which allows not blowing your budget, but remaining generic? Naively, isolating the proprietary interface in a wrapper in your code minimizes changes should you need to move — and blowing your budget isn’t a long term solution. So does it come down to the proprietary option or fail?
- scarface_74 3y ago
  This would only come from someone who has never done or managed a migration at scale. There are so many moving pieces doing any migration at scale, your code is the least of the problems. I have seen it take nearly a year to migrate hundreds of generic VMs with VM hosted databases from on prem to cloud. You can’t get more generic than that. Yes, this was with AWS Professional Services (where I use to work) doing much of the work.
  - zmgsabst 3y ago
    Yawn, insulting others experience to make your point is trite. Perhaps you could justify your point rather than attack other people and appeal to authority?
    - scarface_74 3y ago
      There is so much involved with any migration at scale - data migration, regression testing, network changes even where a company many have a physical connection to their cloud provider, security, compliance , training, and the list goes on.
      - zmgsabst 3y ago
        Okay — and? How is any of that relevant to what I said? You’re ignoring that maintaining your code without modifying your data access (changing only the wrapper) significantly simplifies virtually everything you listed - with the exception of network topology changes.
        scarface_74 3y ago
        Because in the grand scheme of things “your code” and even changing your code is just a minuscule part of what it takes if a migration did ever happen. So the question remains, have you ever been a part of a large scale migration? Have you ever been in the room when the planning was happening?
        zmgsabst 3y ago
        No — changing your code makes every one of those aspects harder, more time consuming, and thus a worse option. (Again, except network topology.) And yes — I’ve helped migrate global systems on the peta-scale, for finance where compliance is legally mandated. The idea that changing your code at the same time is irrelevant is laughable: Changing your software significantly while migrating has caused major errors in projects I’ve seen attempt it — you’re essentially doing a rewrite while trying to migrate, which introduces bugs in compliance, security, etc. - - - - Edit to include a specific example: Let’s imagine I’m migrating a dataset, A. And my copy is A’. In the scenario where I have some wrappers, W and W’, I can write my validation test on the data, T, to work against that interface. So then T(W(A)) = T(W’(A’)) implies that W(A) = W’(A’), since T is a function (and the same code, run both places). My problem in demonstrating my data integrity reduces to ensuring that my wrappers faithfully marshal data — something I’ll get by writing unit tests. In the scenario where I don’t have those data wrappers, I have to write two tests — T and T’. But then what does T(A) = T’(A’) prove? Nothing directly — I need to prove that my two integrity tests are actually measuring the same thing. In this way, isolating the complexity of your data marshaling directly impacts the complexity of downstream tasks, such as verifying the integrity of your data.
scarface_74 3y ago
Please tell how Aurora - which is just MySql and Postgres with a better underlying storage engine - “locks you in”?
- jpdb 3y ago
  I agree with you, but I think the idea is that the underlying storage engine affords you functionality that you can't mimic yourself / elsewhere and therefore your capabilities to leave are limited. You're always "locked in" to some degree and it's an almost worthless thing to invest into if you never actually need to actually migrate off.
athrun 3y ago
You just have to realise that given enough scale everything turns proprietary anyway. The difference being whether you are internally maintaining your special snowflake, or else paying for a solution maintained for you by a 3rd party.

by asah 3y ago

zmgsabst 3y ago

Drive-by response: I’d ask two questions — how possible is it to isolate the code which is proprietary? …is there a third option which allows not blowing your budget, but remaining generic? Naively, isolating the proprietary interface in a wrapper in your code minimizes changes should you need to move — and blowing your budget isn’t a long term solution. So does it come down to the proprietary option or fail?

scarface_74 3y ago
This would only come from someone who has never done or managed a migration at scale. There are so many moving pieces doing any migration at scale, your code is the least of the problems. I have seen it take nearly a year to migrate hundreds of generic VMs with VM hosted databases from on prem to cloud. You can’t get more generic than that. Yes, this was with AWS Professional Services (where I use to work) doing much of the work.
- zmgsabst 3y ago
  Yawn, insulting others experience to make your point is trite. Perhaps you could justify your point rather than attack other people and appeal to authority?
  - scarface_74 3y ago
    There is so much involved with any migration at scale - data migration, regression testing, network changes even where a company many have a physical connection to their cloud provider, security, compliance , training, and the list goes on.
    - zmgsabst 3y ago
      Okay — and? How is any of that relevant to what I said? You’re ignoring that maintaining your code without modifying your data access (changing only the wrapper) significantly simplifies virtually everything you listed - with the exception of network topology changes.
      - scarface_74 3y ago
        Because in the grand scheme of things “your code” and even changing your code is just a minuscule part of what it takes if a migration did ever happen. So the question remains, have you ever been a part of a large scale migration? Have you ever been in the room when the planning was happening?
        zmgsabst 3y ago
        No — changing your code makes every one of those aspects harder, more time consuming, and thus a worse option. (Again, except network topology.) And yes — I’ve helped migrate global systems on the peta-scale, for finance where compliance is legally mandated. The idea that changing your code at the same time is irrelevant is laughable: Changing your software significantly while migrating has caused major errors in projects I’ve seen attempt it — you’re essentially doing a rewrite while trying to migrate, which introduces bugs in compliance, security, etc. - - - - Edit to include a specific example: Let’s imagine I’m migrating a dataset, A. And my copy is A’. In the scenario where I have some wrappers, W and W’, I can write my validation test on the data, T, to work against that interface. So then T(W(A)) = T(W’(A’)) implies that W(A) = W’(A’), since T is a function (and the same code, run both places). My problem in demonstrating my data integrity reduces to ensuring that my wrappers faithfully marshal data — something I’ll get by writing unit tests. In the scenario where I don’t have those data wrappers, I have to write two tests — T and T’. But then what does T(A) = T’(A’) prove? Nothing directly — I need to prove that my two integrity tests are actually measuring the same thing. In this way, isolating the complexity of your data marshaling directly impacts the complexity of downstream tasks, such as verifying the integrity of your data.

scarface_74 3y ago

Please tell how Aurora - which is just MySql and Postgres with a better underlying storage engine - “locks you in”?

jpdb 3y ago
I agree with you, but I think the idea is that the underlying storage engine affords you functionality that you can't mimic yourself / elsewhere and therefore your capabilities to leave are limited. You're always "locked in" to some degree and it's an almost worthless thing to invest into if you never actually need to actually migrate off.

athrun 3y ago

You just have to realise that given enough scale everything turns proprietary anyway. The difference being whether you are internally maintaining your special snowflake, or else paying for a solution maintained for you by a 3rd party.