5 ms·
Much harder to switch cloud providers than to switch LLM models. How much time would it take most companies to move their product from AWS to GCP, for example?
by adam_arthur 3y ago
Much harder to switch cloud providers than to switch LLM models. How much time would it take most companies to move their product from AWS to GCP, for example? What if you use a cloud specific tool like DynamoDB?
Margin is a function of stickiness/cost of switching (among other things).
I suspect eventually we will enter a world where migrating cloud providers is mostly a click of a button, but we're a long ways off from that. Requires vendor agnostic and portable apis/containers/WASM runtimes everywhere.
Swapping an LLM, at least in the current state, is about as close to updating a pointer as you can get.
- throwaway50182 3y agoWell I'd have to change my Terraform provider and the managed Kubernetes resource... Other than that, it'd be the same. So half an hour of coding + half an hour of reconfiguring CI secrets?
- adam_arthur 3y agoPretty obvious to most that switching cloud providers is not quick or painless for the majority of orgs. There's not really an argument in good faith to suggest otherwise. Especially given that many orgs use managed or cloud specific solutions that have no 1:1 mapping between vendors
- throwaway50182 3y agoI said I'd have to change that one managed resource. Actually it's two - the managed database. But no more.
- atmosx 3y ago> I said I'd have to change that one managed resource. Actually it's two - the managed database. But no more. As someone who worked on a similar project recently, I'm getting that idea that you obviously don't know what you're talking about.
- motoxpro 3y agoDidn't you know? It's only one DB ENV variable and you're swapped over from GCP to AWS. /s
- throwaway50182 3y agoI'm running the tech for a global startup with 100M EUR turnover. It's still pretty small, but it's something. You need to plan ahead, sure. Portability was one of my main concerns (including the possibility to go self-hosted). But it's definitely not impossible, nor too hard to do.
- altdataseller 3y agoYou lost me at Terraform and Kubernetes..
- throwaway50182 3y agoWhat's a serious cloud provider-portable replacement for Terraform? And Kubernetes? It goes way beyond containers, you can replace many cloud provider-specific resources with Kubernetes resources. What alternative gives you that?
- intended 3y agoIf you swap an LLM, at the very least, you have to run your entire Eval set again. This will almost certainly lead to a prompt rebuild, to better accommodate new model idiosyncrasies. If you are unlucky- your use case may be one where Evals require human review. Unless you are YOLOing it without evals. In that case this is relevant.
- matusp 3y agoYou need to rerun eval after each LLM update. GPTs have a new version every couple of months and their capabilities can change quite drastically pretty much randomly. Maybe they will make it more robust in time, but I think this is the feature of the technology and people will have to adapt to these quirks
- intended 3y agoYup. Constant Eval.