4 ms·
I wonder if it's RAGing? Dying Internet forums are way, way into the tail of the data.
by jprete 2y ago
I wonder if it's RAGing? Dying Internet forums are way, way into the tail of the data.
- gpderetta 2y agoI asked about gpderetta and he did hallucinate quite a bit (after some flattery).
- atemerev 2y agoNo, I tested it, this is a pure model without RAG (at least no internet search)
- swyx 2y agoplease say what you tested, so that we can understand your effort without necessarily generalizing your conclusions beyond what you actually tried
- atemerev 2y agoAsking for the information about non-public individuals, including myself. RAG-assisted GPT-4 easily provides such information. GPT2 output is consistent with a good model without RAG (it tries to speculate, but says it doesn’t have such information ultimately). I liked that it doesn’t try to hallucinate things.
- zaptrem 2y agoYou could also test this more easily with current events.
- Kuinox 2y agoYou can ask it's knowledge cutoff and it will respond November 2023. It have no idea of the big events of the beginning of 2024, like the earthquake in Japan.
- david_allison 2y agoIt's close to live-ingesting GitHub. I uploaded a text + translation into English on Sunday, and it's almost perfectly replicated the translation. 0 Google results.
- Kuinox 2y agoIt's unable to know anything about the project I work on, so I don't think so.
- swyx 2y agohow is this proof of live ingesting github?
- brianjking 2y agolol, clearly doesn't seem to be.
- Jensson 2y ago> You can ask it's knowledge cutoff and it will respond November 2023 It probably just repeated something based on what common AI cutoffs there are, LLMs doesn't have a sense or self or thought process, they don't know more about themselves than the text given to them about themselves, and even then it is likely to default to some common text from the internet.
- jstanley 2y agoI think it's prompted with a bunch of context information (like, "you are a helpful, harmless, honest AI assistant created by OpenAI, your knowledge cutoff date is ..., please answer the user's questions"). If you really think it is just saying whatever it read on the web, how do you explain that not all LLM chatbots claim to be ChatGPT? Engineering is happening, it's not just a raw model of text from the web connected directly to the user.
- Kuinox 2y ago
- hombre_fatal 2y agoWhat database would it RAG from? The long tail in the model's data is also the long tail of any data. There are no google searches that have the information it provided about my forum nor is the info neatly collated anywhere on my own forum. Its knowledge about my forum isn't only obscure, it's also an itemization of obscure events over time to draw obscure conclusions that only a historian of my forum would know. That's what's so impressive. Granted, my forum was once the largest forum of its genre though that was almost 15 years ago, so it's not some dead proboards forum.