Most models ("AI" or otherwise) behave very poorly in the situation known a "structural break" [1]. In extreme instances you can even end up with something like

Most models ("AI" or otherwise) behave very poorly in the situation known a "structural break" [1]. In extreme instances you can even end up with something like Simpsons paradox where the predicted correlation is the opposite of the true correlation [2]. I've had to deal with this exact problem before. The trick is to have a meta model that sits on top and tries to detect when the recent data is highly unlikely to have been produced if the existing model is true. When that happens you flush the older data. This might be what our brains do [3]. [1] https://en.wikipedia.org/wiki/Structural_break https://en.wikipedia.org/wiki/Structural_break [2] https://en.wikipedia.org/wiki/Simpson%27s_paradox https://en.wikipedia.org/wiki/Simpson%27s_paradox [3] https://slatestarcodex.com/2017/09/06/predictive-processing-and-perceptual-control/ https://slatestarcodex.com/2017/09/06/predictive-processing-...