4 ms·
Please can someone explain the gist of this article to a mere human with no PhD in AI?
by JPLeRouzic 3y ago
Please can someone explain the gist of this article to a mere human with no PhD in AI?
- JPLeRouzic 3y agoI asked ChatGPT, and it reluctantly told: "The phenomena discussed in the text are more about the inherent characteristics of the model's representation space rather than the input data being flawed." It took three attempts, the first two were merely a rephrasing/paraphrasing of the abstract.
- ta988 3y agoTokens (parts of words, words, symbols) are represented as vectors. They say there are nokens which are vectors that seem to represent something because they are organized in that space in a regular manner. They think those vectors may have something interesting about them because they seem to be somewhat organized and related to each other.
- ta988 3y agoAlso the seem to correspond to things that exist in our world and curiously like things that have existed for a long time, not too recent things.
- JPLeRouzic 3y agoPlease could you point out where these strange definitions are found in the article?
- great_psy 3y agoSearch for the string “children” in the context of children books/stories
- JPLeRouzic 3y agoThanks Great_psy.
- lainga 3y agoLindySpace?
- JPLeRouzic 3y agoThanks for taking care to explain. But aren't all embedded vectors representing something organized?
- ta988 3y ago"This is described, and then the remaining expanse of the embedding space is explored by using simple prompts to elicit definitions for non-token custom embedding vectors (so-called "nokens")" so they make new vectors
- netruk44 3y agoAt a high level: It's an article observing how GPT-J token embeddings are positioned in 'embedding space', with some connections drawn to GPT 3. They then experiment with having GPT-J provide definitions for "nokens" (A "noken" is basically a made up embedding created by modifying the embedding generated from a real token) to see what happens when a model is presented with a novel embedding outside of the trained embedding space to see how it interprets them. Diving in a little more: An observation from the article is that the first letter of a word represented by an embedding is actually encoded into that embedding such that you could identify what letter any embedding starts with, with 98% certainty. They observe this linear relationship and devise an experiment. They ask GPT-J what letter the word "icon" starts with, and it correctly replies "I". They then create a "noken" for the word "icon" and modify it so that it no longer represents that the word starts with "I" and ask GPT-J again what letter this "noken" starts with. GPT-J then incorrectly replies that the word "icon" does not start with the letter "I". So then they pose the question "can the model still define the word 'broccoli' even if we shift the first letter away from 'B'?" and the answer is almost always yes, changing the semantic "first letter" of an embedding doesn't change the model's ability to understand the embedding. They then do a series of experiments asking GPT-J to provide the "typical definition for the word '<noken>'." with the example word 'hate' being used. They gradually shifted the first letter away from 'H', and see that up until very high levels of modification, the model can still give a normal definition for the modified 'hate' noken ("a strong feeling of dislike or hostility."). At extreme modifications to the noken, the model starts providing strange definitions ("a person who is not a member of a particular group.", "a period of time during which a person or thing is in a state of being"). They repeat the experiment and note that this behavior of the definition keeping stable up until 'collapse' is common across many tokens: > ...usually ending up with something about a person who isn't a member of a group by k = 100, having passed through one or more other themes involving things like Royal families, places of refuge, small round holes and yellowish-white things, to name a few of the baffling tropes that began to appear regularly.
- uoaei 3y agoDon't believe everything you read on the internet just because it's written in an authoritative voice on a website you've heard of.
- JPLeRouzic 3y agoAh, ah, I agree!
- wing-_-nuts 3y agoI try to keep up with the space over on r/locallamma and I've never seen such a word salad. I was convinced the author was writing satire until I got a few paragraphs in and it just ...didn't stop.
- JPLeRouzic 3y agoThanks that's my impression also: For me, it seems they did not discover a general feature of LLM, rather they discovered a feature they have introduced in their instance of GTP-J. That said I am a noob about LLMs.