"ChatGPT and the likes do not have a memory" Can't we consider the context window to be memory? And eventually wont we have larger and larger context windows,

"ChatGPT and the likes do not have a memory" Can't we consider the context window to be memory? And eventually wont we have larger and larger context windows, and perhaps even individual training where part of the context window 'conversation' is also fed back into the training data? Seems like this will come to pass eventually.