3 ms·

Did you generate a bunch all at once before starting to get some idea of what the natural variance looks like? I would think it's important to verify some level

by sp0rk 3y ago

Did you generate a bunch all at once before starting to get some idea of what the natural variance looks like? I would think it's important to verify some level of progression over time, because with the current four it seems entirely possible that the examples could have all been generated at the same time with no changes to the model.

ratg13 3y ago
Aren’t they using the March 14 model like the general public? It’s frozen in time, there are no updates to it.. All of these will be drawn using the same model until they push a new update, or you switch to a different GPT But I already think they proved the point that the generation is random enough that it would be extremely difficult to track progress this way.
- williamstein 3y ago
  GPT's output is by default somewhat random. If you ask the same exact question several times, you'll potentially get several different answers. Each successive word in the output is chosen from a distribution of possibilities -- that distribution is fixed, but that actual sample chosen from the distribution is not fixed. See, e.g., https://platform.openai.com/docs/api-reference/completions/create#completions/create-temperature https://platform.openai.com/docs/api-reference/completions/c...
  - vorticalbox 3y ago
    When using the openAi end points you can set the temperature, k_top etc. That will allow you to tone down the randomness. Passing a temperature of 0 would mean same input always has the same output.
- int_19h 3y ago
  When I ask GPT-4 to draw a unicorn, I get this: https://i.imgur.com/HQ1Vjp0.png https://i.imgur.com/HQ1Vjp0.png And when I ask it to draw Mona Lisa, I get this: https://i.imgur.com/gsBU49w.png https://i.imgur.com/gsBU49w.png So far as I can see, the only difference is that the GitHub repo uses the API; the ones that I have were rendered using the web chat UI. Which makes me wonder if they're using the bleeding edge model for the chat.
  - imdsm 3y ago
    That's a really interesting question! I haven't yet gained access to the enhanced chat features with image outputs. I'm using the API with default parameters, with the gpt-4-0314 model, outputting SVG. It's not a perfect experiment, but we'll see how it gets on over time.
    - int_19h 3y ago
      To be clear, I'm not using multimodal chat, either. I just told it to output an SVG snippet in the chat and then manually copy/pasted it to view.
gwern 3y ago
Also unclear if he's sampling at temp=0. Looks like he doesn't set a temp? https://github.com/adamkdean/gpt-unicorn/blob/8ad76ec7161682af531f0b23ab14f4fdbfaf41f5/src/lib/chatapi.js#L7 https://github.com/adamkdean/gpt-unicorn/blob/8ad76ec7161682... So not sure what he's really doing.
- capableweb 3y ago
  Then it's using the default value which is temperature=1.0, which is by no way deterministic (not that temperature=0.0 is either, but it's more likely to give similar responses to similar prompts, than 1.0 is)

by sp0rk 3y ago

ratg13 3y ago

Aren’t they using the March 14 model like the general public? It’s frozen in time, there are no updates to it.. All of these will be drawn using the same model until they push a new update, or you switch to a different GPT But I already think they proved the point that the generation is random enough that it would be extremely difficult to track progress this way.

williamstein 3y ago
GPT's output is by default somewhat random. If you ask the same exact question several times, you'll potentially get several different answers. Each successive word in the output is chosen from a distribution of possibilities -- that distribution is fixed, but that actual sample chosen from the distribution is not fixed. See, e.g., https://platform.openai.com/docs/api-reference/completions/create#completions/create-temperature https://platform.openai.com/docs/api-reference/completions/c...
- vorticalbox 3y ago
  When using the openAi end points you can set the temperature, k_top etc. That will allow you to tone down the randomness. Passing a temperature of 0 would mean same input always has the same output.
int_19h 3y ago
When I ask GPT-4 to draw a unicorn, I get this: https://i.imgur.com/HQ1Vjp0.png https://i.imgur.com/HQ1Vjp0.png And when I ask it to draw Mona Lisa, I get this: https://i.imgur.com/gsBU49w.png https://i.imgur.com/gsBU49w.png So far as I can see, the only difference is that the GitHub repo uses the API; the ones that I have were rendered using the web chat UI. Which makes me wonder if they're using the bleeding edge model for the chat.
- imdsm 3y ago
  That's a really interesting question! I haven't yet gained access to the enhanced chat features with image outputs. I'm using the API with default parameters, with the gpt-4-0314 model, outputting SVG. It's not a perfect experiment, but we'll see how it gets on over time.
  - int_19h 3y ago
    To be clear, I'm not using multimodal chat, either. I just told it to output an SVG snippet in the chat and then manually copy/pasted it to view.

gwern 3y ago

Also unclear if he's sampling at temp=0. Looks like he doesn't set a temp? https://github.com/adamkdean/gpt-unicorn/blob/8ad76ec7161682af531f0b23ab14f4fdbfaf41f5/src/lib/chatapi.js#L7 https://github.com/adamkdean/gpt-unicorn/blob/8ad76ec7161682... So not sure what he's really doing.

capableweb 3y ago
Then it's using the default value which is temperature=1.0, which is by no way deterministic (not that temperature=0.0 is either, but it's more likely to give similar responses to similar prompts, than 1.0 is)