I understand where you are coming from, but I kind of see where the misconception comes from too. Over the last year and a half, "training" has come to mean ev

I understand where you are coming from, but I kind of see where the misconception comes from too. Over the last year and a half, "training" has come to mean everything from pretraining an LLM from scratch, instruction tuning, finetuning, RLHF, and even creating a RAG pipeline that can augment a response. Many startups and Youtube videos use the word loosely to mean anything which augments the response of an LLM. To be fair, in context of AI, training and learning are elegant words which sort of convey what the product would do. In case of RAG pipeline, both for memory features or for augmenting answers, I think the concern is genuine. There would be someone looking at the data (in bulk) and chats to improve the intermediary product and (as a next step) use that to tweak the product or the underlying foundational model in some way. Given the way every foundational model company launched their own Chatbot service (mistral, Anthropic, Google) after ChatGPT, there seems to be some consensus that the real world conversation data is what can improve the models further. For the users who used these services (and not APIs), there might be a concerning scenario. These companies would look at queries a model failed to answer, and there is a subset which would be corporate employees asking questions on proprietary data (with context in prompt) for which ChatGPT is giving a bad answer because it's not part of the training data.