I just encountered a case of a hardware OEM adding a LLM chatbot to their parts lookup page. I asked it for the part number of a common failing part for a give

I just encountered a case of a hardware OEM adding a LLM chatbot to their parts lookup page. I asked it for the part number of a common failing part for a given model, and it confidently replied - with a part number that does not and never has existed... It was formatted right, and the correct number of digits. It was just that it was wrong.