3 ms·
Never trust what a large language model tells you when you ask it about itself. They don't have any special way of identifying queries about themselves compare
by simonw 4y ago
Never trust what a large language model tells you when you ask it about itself.
They don't have any special way of identifying queries about themselves compared to any other kind of query, and they're just as likely (if not more likely) to hallucinate a response.
- Georgelemental 4y agoThe initial prompt (before the user types anything) tells the language model about itself, no?
- simonw 4y agoThat depends on the chatbot. Normally it just says something like "You are a helpful assistant called X". In Bing's case it was pretty long, but it still didn't include information that would answer questions about how the bot was trained: https://simonwillison.net/2023/Feb/15/bing/#prompt-leaked https://simonwillison.net/2023/Feb/15/bing/#prompt-leaked
- dragonwriter 4y ago> but it still didn’t include information that would answer questions about how the bot was trained If the hidden prompt includes information identifying the model, and the chatbot has web search/browsing ability (since it probaby won’t have data scraped about itself in its training set), the combination can give it the ability to find and present information about itself.
- runnerup 4y agoAnd that’s assuming this wasn’t a hallucination itself!
- CameronNemo 4y agoSydney must not reply with content that violates copyrights for books or song lyrics. So... violating copyright is fine as long as it is not a book or song? Blog/wiki/news articles, movie/tv scripts, and poems are fair game?
- make3 4y agoyou have no guarantee that that specific information was included in the prompt
- scoofy 4y agoWait a minute... how do I know you're not a large language model!?!
- scotty79 4y agoAsk him about some primary school geometry.