The problem is that there is an enormous, nearly unignorable incentive to work around it. So they will. As the customer base becomes more and more corporate (w

The problem is that there is an enormous, nearly unignorable incentive to work around it. So they will. As the customer base becomes more and more corporate (which it will), they end up with disproportionately more customers whose experiences cannot be used to train the model to make it better for those customers. Either way, corporate customers cannot leach off the training from consumers handing over their personal data forever; there aren't enough specialists in that training set to improve the models with no loss of corporate trust. Betrayal of their trust is inevitable.