7 ms·

I'm more curious how they think LLM's can imagine things: > To understand symbolic programs, LLMs may need to possess the ability to imagine how the correspond

I'm more curious how they think LLM's can imagine things: > To understand symbolic programs, LLMs may need to possess the ability to imagine how the corresponding graphics content would look without directly accessing the rendered visual content To my understanding, LLMs are predictive engines based upon their tokens and embeddings without any ability to "imagine" things. As such, an LLM might be able to tell you that the following SVG is a black circle because it is in Mozilla documentation[0]: <svg viewBox="0 0 100 100" xmlns="http://www.w3.org/2000/svg"> <circle cx="50" cy="50" r="50" /> </svg> However, I highly doubt any LLM could tell you the following is a "Hidden Mickey" or "Mickey Mouse Head Silhouette": <svg viewBox="0 0 175 175" xmlns="http://www.w3.org/2000/svg"> <circle cx="100" cy="100" r="50" /> <circle cx="50" cy="50" r="40" /> <circle cx="150" cy="50" r="40" /> </svg> - [0] https://developer.mozilla.org/en-US/docs/Web/SVG/Element/circle https://developer.mozilla.org/en-US/docs/Web/SVG/Element/cir...

randomdata 2y ago
> without any ability to "imagine" things. What's imagining, then? The way LLMs explore different predictive branches in order to find an optimal solution doesn't seem all that different than what I consider imagining: Thinking about what could be and considering different variations on that idea. An LLM isn't a brain, so there is no implication of it being said in the truest human sense, but it seems like a decent analogy to me.
- merlincorey 2y ago
  As I mentioned elsewhere, all the modern models are multimodal models so I think it would be fair to say rendering the SVG and then having the dedicated image model classify it is similar to "imagining", but it is definitely not strictly part of a Large Language Model which only predicts the next token based on previous tokens.
rel_ic 2y ago
Check out Act One of this This American Life episode https://www.thisamericanlife.org/803/transcript https://www.thisamericanlife.org/803/transcript TLDR: it seems like an LLM might be able to tell you your SVG is a "Mickey Mouse Head Silhouette"
- kgen 2y ago
  I was just about to post the same thing -- quite a fascinating test of gpt's capabilities
montebicyclelo 2y ago
Chat GPT: > Given the arrangement of three overlapping circles, it resembles the classic depiction of a *Mickey Mouse* head silhouette: > The two smaller circles represent Mickey's ears. > The larger circle represents his head. > This is a stylized version of the iconic Mickey Mouse logo. Imo: In order to predict the next token for non-trivial tokens, (of which there are many on the training data), you do have to do some more complex thinking/reasoning than just a lookup of past training data.
- Lichtso 2y ago
  Can you challenge it and e.g. double the radius of the last circle? That should make the rendered output unintelligible. Let's see if it still thinks it represents the iconic logo.
  - jaymzcampbell 2y ago
    Changing the last one to r=80 instead of r=40, GPT4o ends up saying: > The circles overlap in certain areas, depending on their size and position, creating a layered visual effect where the largest circle (third one) dominates most of the canvas space. Though tbf to it, I'm not sure I'd say this looks like MM either: https://i.imgur.com/0VHdocf.png https://i.imgur.com/0VHdocf.png (unless I knew this was the intent prior).
  - montebicyclelo 2y ago
    It passes the test you've proposed. For the below results, the prompt is the same for A and B, except for in B the radius of the last circle is twice as big to make the rendered output unintelligible. A. Valid Mickey is detected by the model. "...This arrangement might resemble a basic version of a Mickey Mouse shape, where the two smaller circles represent the ears and the larger circle represents the head...", https://chatgpt.com/share/3999859a-b6db-4671-8b69-0ec6a5bac398 https://chatgpt.com/share/3999859a-b6db-4671-8b69-0ec6a5bac3... B. Invalid Mickey is not misclassified as Mickey by the model and is correctly described. "...these circles will overlap, creating a pattern where the largest circle (Circle 3) dominates the right side of the canvas, with the other two smaller circles overlapping it and each other in the middle...", https://chatgpt.com/share/df3c57ac-495b-4e4c-b00c-bae31781c4de https://chatgpt.com/share/df3c57ac-495b-4e4c-b00c-bae31781c4...
jaymzcampbell 2y ago
GPT4o certainly gave it a good go (prompt: "What cartoon character would you say this SVG code renders most closely to?"). That's maybe a little too "prompting" tbf: """ The SVG code you provided creates an image with three circles: A large circle at the center (at coordinates 100, 100) with a radius of 50. Two smaller circles at the top left (at coordinates 50, 50) and top right (at coordinates 150, 50) with a radius of 40 each. This configuration closely resembles the head and ears of Mickey Mouse. The larger circle represents the head, and the two smaller circles represent the ears. """ Slightly more generic prompt: "What is this code drawing? Can you suggest what it might be representing?" """ What It Might Represent: Mickey Mouse Head: The arrangement resembles a simplified "Mickey Mouse" head, where the large central circle represents the head, and the two smaller circles represent the ears. Abstract Art: Alternatively, it could be an abstract design with three overlapping circles of different sizes, suggesting some form of simple geometric art. The circles are positioned such that the smaller ones are symmetrically placed above the larger central one, which is a typical representation of a head with ears. """ And more generic "I've been sent this code, what is it drawing?" ends with: """ You will see three overlapping circles. The largest circle is near the center, while two smaller circles are positioned above and slightly to the left and right of the center. The arrangement may resemble a basic representation of a Mickey Mouse head shape, with the large circle as the face and the two smaller circles as ears. """
- CaptainFever 2y ago
  I'm impressed. How does it do that?
- kevindamm 2y ago
  Were those all within the same conversation, or the beginnings of separate conversations?
  - jaymzcampbell 2y ago
    Separate conversations, I wanted to avoid any prehistory, though all under my logged in user so can't exclude any possible inference that way
- 2y ago
deleted 2y ago
[deleted]
westurner 2y ago
If teh LLM saves the SVG vector graphic to a raster image like a PNG and prompts with that instead, it will have no trouble labeling what's depicted in the SVG. So, the task is "describe what an SVG depicts without saving it to a raster image and prompting with that"?
- merlincorey 2y ago
  I believe this is exactly what the modern multimodal models are doing -- they are not strictly Large Language Models any more.
  - westurner 2y ago
    Which part is the query preprocessor? /? LLM stack app architecture https://www.google.com/search?q=LLM+stack+app+architecture&tbm=isch https://www.google.com/search?q=LLM+stack+app+architecture&t... https://cobusgreyling.medium.com/emerging-large-language-model-llm-application-architecture-cba0e7862037 https://cobusgreyling.medium.com/emerging-large-language-mod... ; Flexibility / Complexity,
ctoth 2y ago
> However, I highly doubt I never understand why people don't just test this? Claude 3.5 sonnet, first test, prompt: Any idea what this SVG is describing?: This SVG describes a simple geometric composition consisting of three circles: 1. The largest circle is centered at coordinates (100, 100) with a radius of 50 units. 2. A smaller circle is positioned at the top-left, centered at (50, 50) with a radius of 40 units. 3. Another smaller circle is positioned at the top-right, centered at (150, 50), also with a radius of 40 units. The overall effect, if rendered, would resemble a basic face-like structure or a simplified Mickey Mouse silhouette, with two smaller circles (potentially representing ears or eyes) above a larger central circle (potentially representing a face or head). The SVG uses a viewBox of "0 0 175 175", which defines the coordinate system and aspect ratio for the image. Would you like me to create a visual representation of this SVG for you?
- deleted 2y ago
  [deleted]
- smallnamespace 2y ago
  > I never understand why people don't just test this? One reason might be that you get the benefit of feeling superior to LLMs without any cost or risk of finding yourself mistaken.
  - merlincorey 2y ago
    [flagged]
- merlincorey 2y ago
  > I never understand why people don't just test this? I never understand why people take things out of context instead of responding to the totality of a message, since I started off by saying: > To my understanding, You have responded not by providing your understanding but by belittling me. Nonetheless, I appreciate you taking the time to query Claude-3.5 and sharing the results. According to Amazon Bedrock adds Claude 3.5-sonnect[0] is a multimodal model with state-of-the-art vision processing, which tells me that Claude can "cheat" by rendering the SVG and then passing to the image model for the results which is perhaps one way of describing "imagining" things, but it does not describe a strict Large Language Model which predicts the next token. > The multimodal Claude 3.5 Sonnet also excels at processing images with state-of-the-art vision, particularly when interpreting charts and graphs—helping to get faster, deeper insights from data. It can accurately decipher text from imperfect images—for example, poorly scanned documents—and in doing so, glean more insights than from text alone. [0] https://www.aboutamazon.com/news/aws/amazon-bedrock-anthropic-ai-claude-3-5-sonnet https://www.aboutamazon.com/news/aws/amazon-bedrock-anthropi...
deleted 2y ago
[deleted]

randomdata 2y ago

> without any ability to "imagine" things. What's imagining, then? The way LLMs explore different predictive branches in order to find an optimal solution doesn't seem all that different than what I consider imagining: Thinking about what could be and considering different variations on that idea. An LLM isn't a brain, so there is no implication of it being said in the truest human sense, but it seems like a decent analogy to me.

merlincorey 2y ago
As I mentioned elsewhere, all the modern models are multimodal models so I think it would be fair to say rendering the SVG and then having the dedicated image model classify it is similar to "imagining", but it is definitely not strictly part of a Large Language Model which only predicts the next token based on previous tokens.

rel_ic 2y ago

Check out Act One of this This American Life episode https://www.thisamericanlife.org/803/transcript https://www.thisamericanlife.org/803/transcript TLDR: it seems like an LLM might be able to tell you your SVG is a "Mickey Mouse Head Silhouette"

kgen 2y ago
I was just about to post the same thing -- quite a fascinating test of gpt's capabilities

montebicyclelo 2y ago

Chat GPT: > Given the arrangement of three overlapping circles, it resembles the classic depiction of a *Mickey Mouse* head silhouette: > The two smaller circles represent Mickey's ears. > The larger circle represents his head. > This is a stylized version of the iconic Mickey Mouse logo. Imo: In order to predict the next token for non-trivial tokens, (of which there are many on the training data), you do have to do some more complex thinking/reasoning than just a lookup of past training data.

Lichtso 2y ago
Can you challenge it and e.g. double the radius of the last circle? That should make the rendered output unintelligible. Let's see if it still thinks it represents the iconic logo.
- jaymzcampbell 2y ago
  Changing the last one to r=80 instead of r=40, GPT4o ends up saying: > The circles overlap in certain areas, depending on their size and position, creating a layered visual effect where the largest circle (third one) dominates most of the canvas space. Though tbf to it, I'm not sure I'd say this looks like MM either: https://i.imgur.com/0VHdocf.png https://i.imgur.com/0VHdocf.png (unless I knew this was the intent prior).
- montebicyclelo 2y ago
  It passes the test you've proposed. For the below results, the prompt is the same for A and B, except for in B the radius of the last circle is twice as big to make the rendered output unintelligible. A. Valid Mickey is detected by the model. "...This arrangement might resemble a basic version of a Mickey Mouse shape, where the two smaller circles represent the ears and the larger circle represents the head...", https://chatgpt.com/share/3999859a-b6db-4671-8b69-0ec6a5bac398 https://chatgpt.com/share/3999859a-b6db-4671-8b69-0ec6a5bac3... B. Invalid Mickey is not misclassified as Mickey by the model and is correctly described. "...these circles will overlap, creating a pattern where the largest circle (Circle 3) dominates the right side of the canvas, with the other two smaller circles overlapping it and each other in the middle...", https://chatgpt.com/share/df3c57ac-495b-4e4c-b00c-bae31781c4de https://chatgpt.com/share/df3c57ac-495b-4e4c-b00c-bae31781c4...

jaymzcampbell 2y ago

GPT4o certainly gave it a good go (prompt: "What cartoon character would you say this SVG code renders most closely to?"). That's maybe a little too "prompting" tbf: """ The SVG code you provided creates an image with three circles: A large circle at the center (at coordinates 100, 100) with a radius of 50. Two smaller circles at the top left (at coordinates 50, 50) and top right (at coordinates 150, 50) with a radius of 40 each. This configuration closely resembles the head and ears of Mickey Mouse. The larger circle represents the head, and the two smaller circles represent the ears. """ Slightly more generic prompt: "What is this code drawing? Can you suggest what it might be representing?" """ What It Might Represent: Mickey Mouse Head: The arrangement resembles a simplified "Mickey Mouse" head, where the large central circle represents the head, and the two smaller circles represent the ears. Abstract Art: Alternatively, it could be an abstract design with three overlapping circles of different sizes, suggesting some form of simple geometric art. The circles are positioned such that the smaller ones are symmetrically placed above the larger central one, which is a typical representation of a head with ears. """ And more generic "I've been sent this code, what is it drawing?" ends with: """ You will see three overlapping circles. The largest circle is near the center, while two smaller circles are positioned above and slightly to the left and right of the center. The arrangement may resemble a basic representation of a Mickey Mouse head shape, with the large circle as the face and the two smaller circles as ears. """

CaptainFever 2y ago
I'm impressed. How does it do that?
kevindamm 2y ago
Were those all within the same conversation, or the beginnings of separate conversations?
- jaymzcampbell 2y ago
  Separate conversations, I wanted to avoid any prehistory, though all under my logged in user so can't exclude any possible inference that way
2y ago

deleted 2y ago

[deleted]

westurner 2y ago

If teh LLM saves the SVG vector graphic to a raster image like a PNG and prompts with that instead, it will have no trouble labeling what's depicted in the SVG. So, the task is "describe what an SVG depicts without saving it to a raster image and prompting with that"?

merlincorey 2y ago
I believe this is exactly what the modern multimodal models are doing -- they are not strictly Large Language Models any more.
- westurner 2y ago
  Which part is the query preprocessor? /? LLM stack app architecture https://www.google.com/search?q=LLM+stack+app+architecture&tbm=isch https://www.google.com/search?q=LLM+stack+app+architecture&t... https://cobusgreyling.medium.com/emerging-large-language-model-llm-application-architecture-cba0e7862037 https://cobusgreyling.medium.com/emerging-large-language-mod... ; Flexibility / Complexity,

ctoth 2y ago

> However, I highly doubt I never understand why people don't just test this? Claude 3.5 sonnet, first test, prompt: Any idea what this SVG is describing?: This SVG describes a simple geometric composition consisting of three circles: 1. The largest circle is centered at coordinates (100, 100) with a radius of 50 units. 2. A smaller circle is positioned at the top-left, centered at (50, 50) with a radius of 40 units. 3. Another smaller circle is positioned at the top-right, centered at (150, 50), also with a radius of 40 units. The overall effect, if rendered, would resemble a basic face-like structure or a simplified Mickey Mouse silhouette, with two smaller circles (potentially representing ears or eyes) above a larger central circle (potentially representing a face or head). The SVG uses a viewBox of "0 0 175 175", which defines the coordinate system and aspect ratio for the image. Would you like me to create a visual representation of this SVG for you?

deleted 2y ago
[deleted]
smallnamespace 2y ago
> I never understand why people don't just test this? One reason might be that you get the benefit of feeling superior to LLMs without any cost or risk of finding yourself mistaken.
- merlincorey 2y ago
  [flagged]
merlincorey 2y ago
> I never understand why people don't just test this? I never understand why people take things out of context instead of responding to the totality of a message, since I started off by saying: > To my understanding, You have responded not by providing your understanding but by belittling me. Nonetheless, I appreciate you taking the time to query Claude-3.5 and sharing the results. According to Amazon Bedrock adds Claude 3.5-sonnect[0] is a multimodal model with state-of-the-art vision processing, which tells me that Claude can "cheat" by rendering the SVG and then passing to the image model for the results which is perhaps one way of describing "imagining" things, but it does not describe a strict Large Language Model which predicts the next token. > The multimodal Claude 3.5 Sonnet also excels at processing images with state-of-the-art vision, particularly when interpreting charts and graphs—helping to get faster, deeper insights from data. It can accurately decipher text from imperfect images—for example, poorly scanned documents—and in doing so, glean more insights than from text alone. [0] https://www.aboutamazon.com/news/aws/amazon-bedrock-anthropic-ai-claude-3-5-sonnet https://www.aboutamazon.com/news/aws/amazon-bedrock-anthropi...

deleted 2y ago

[deleted]