6 ms·

"The generated code might not always be correct. In that case, run it again lmao" is the best documentation I've read all week.

arecurrence 4y ago
Copilot has an option (maybe beta users only) to see alternative generations for a prompt. It's really handy because some generations are one-liners while others are entire functions. Perhaps this is infeasible with the current cost per generation but a multiple choice display could be handy (perhaps split the screen into quadrants and pick the one that most fits what you prefer).
- Veen 4y ago
  Google Bard has a similar feature, and you can ask the OpenAI API to generate multiple completions and pick the best one (best_of) or return some number of the generated completions (n). In OpenAI's case, best of means "the one with the highest log probability per token".
junon 4y ago
If you watch the video (I recommend at 0.5x speed) the author adds "you feel me?" to one of the prompts and I laughed.
Spivak 4y ago
Hey everyone makes mistakes when coding, even LLms. There’s two approaches I use, I’m sure there are more. * Do multiple completions and, filter the ones that successfully run, and take the most common result. * Do a completion, if it fails, ask the Llm to find and correct the bug.
- chaxor 4y ago
  This sounds like reducing the temperature with more steps
  - toxicFork 4y ago
    Reducing temperature doesn't automatically result in correctness, it only results in more precision. It could be inaccurate with high precision - like a bow that consistently undershoots, and you would have a harder time trying to correct it.
    - chaxor 4y ago
      You're right, and I understand that - but you're less likely to get the same result with high temperature, so it will be hard to find correct outputs when using overlap of output. But combining pieces that look good to you of several outputs is a good way to use it. Much of the natural language it creates is useful to me as essentially an extended thesaurus, or to make slightly different points I hadn't thought of it emails. It undoubtedly (demonstrably and probably, I'm sure) knows *way* more than any human on Earth, so it's great at teaching people new things. I usually just reword everything slightly that GPTs provide, so it's a *phenomenal* learning tool.
carlsborg 4y ago
From authors website: "I graduated high school in May of 2022..". Elsewhere, he writes he placed first at a university level drone programming challenge at IIT-Bombay Techfest. Very impressive.
- quonn 4y ago
  The JEE exam in India is very advanced, I doubt most Europeans or US students could pass it, even after a year or two at university. So those who enter IIT are already quite good very early.
  - jsdeveloper 4y ago
    IIT colleges in themselves are not even in top 100 Universities of the world. If you train a lion in a circus, it won't be the king of the jungle. So if we were to believe your statement: > I doubt most Europeans or US students could pass it, even after a year or two at university. than this allegedly miraclous students are wasting there talents studying in IITs!
    - gcr 4y ago
      USA and European university rankings have a strong bias against universities in the Global South (especially India), in part because of the stigma against Indian programmers and in part because they're generally wanting to promote US and European universities as the global leaders. If you're going to disparage a university based on nothing but ranking lists run by USA and Europeans, you're missing out. IIT students are amazing, i've worked with them. Far more creative and competent programmers than the average Ivy League grad in my cohort.
      - tekknik 4y ago
        So the racism card again? Why would two independent governing bodies group together to oppress a single country? Would citizens of the EU and US not want to go to one of the best schools regardless of locale?
newhouseb 4y ago
We also use GPT to perform actions in the software I build at work and we hit the same issue of inconsistency which lead me down a long rabbit hole to see if I could force an LLM to only emit grammatically correct output that follows a bespoke DSL (a DSL ideally safer and more precise than just eval'ing random AI-produced Python). I just finished writing up a long post [1] that describes how this can work on local models. It's a bit tricky to do via API efficiently, but hopefully OpenAI will give us the primitives one day to appropriately steer these models to do the right things (tm). [1] https://github.com/newhouseb/clownfish https://github.com/newhouseb/clownfish
elif 4y ago
I've been building a ChatGPT project this week and this explanation is so true it made me actually lol. Sometimes it's impressively flawless, sometimes it gets stuck in broken patterns... Even if the inputs are identical. "Monkey around with the prompt and pray" has replaced unit testing.
- OOPMan 4y ago
  Ooof
- MilStdJunkie 4y ago
  Exactly my experience. I've asked for some time metrics to see if the repeat is actually taking longer than the work.
csmpltn 4y ago
Why aren't they collecting feedback signals from people on the results produced by the model? "This was correct", "this was garbage", etc.
- reportgunner 4y ago
  What if someone was asking to generate garbage in Blender though?

arecurrence 4y ago

Copilot has an option (maybe beta users only) to see alternative generations for a prompt. It's really handy because some generations are one-liners while others are entire functions. Perhaps this is infeasible with the current cost per generation but a multiple choice display could be handy (perhaps split the screen into quadrants and pick the one that most fits what you prefer).

Veen 4y ago
Google Bard has a similar feature, and you can ask the OpenAI API to generate multiple completions and pick the best one (best_of) or return some number of the generated completions (n). In OpenAI's case, best of means "the one with the highest log probability per token".

junon 4y ago

If you watch the video (I recommend at 0.5x speed) the author adds "you feel me?" to one of the prompts and I laughed.

Spivak 4y ago

Hey everyone makes mistakes when coding, even LLms. There’s two approaches I use, I’m sure there are more. * Do multiple completions and, filter the ones that successfully run, and take the most common result. * Do a completion, if it fails, ask the Llm to find and correct the bug.

chaxor 4y ago
This sounds like reducing the temperature with more steps
- toxicFork 4y ago
  Reducing temperature doesn't automatically result in correctness, it only results in more precision. It could be inaccurate with high precision - like a bow that consistently undershoots, and you would have a harder time trying to correct it.
  - chaxor 4y ago
    You're right, and I understand that - but you're less likely to get the same result with high temperature, so it will be hard to find correct outputs when using overlap of output. But combining pieces that look good to you of several outputs is a good way to use it. Much of the natural language it creates is useful to me as essentially an extended thesaurus, or to make slightly different points I hadn't thought of it emails. It undoubtedly (demonstrably and probably, I'm sure) knows *way* more than any human on Earth, so it's great at teaching people new things. I usually just reword everything slightly that GPTs provide, so it's a *phenomenal* learning tool.

carlsborg 4y ago

From authors website: "I graduated high school in May of 2022..". Elsewhere, he writes he placed first at a university level drone programming challenge at IIT-Bombay Techfest. Very impressive.

quonn 4y ago
The JEE exam in India is very advanced, I doubt most Europeans or US students could pass it, even after a year or two at university. So those who enter IIT are already quite good very early.
- jsdeveloper 4y ago
  IIT colleges in themselves are not even in top 100 Universities of the world. If you train a lion in a circus, it won't be the king of the jungle. So if we were to believe your statement: > I doubt most Europeans or US students could pass it, even after a year or two at university. than this allegedly miraclous students are wasting there talents studying in IITs!
  - gcr 4y ago
    USA and European university rankings have a strong bias against universities in the Global South (especially India), in part because of the stigma against Indian programmers and in part because they're generally wanting to promote US and European universities as the global leaders. If you're going to disparage a university based on nothing but ranking lists run by USA and Europeans, you're missing out. IIT students are amazing, i've worked with them. Far more creative and competent programmers than the average Ivy League grad in my cohort.
    - tekknik 4y ago
      So the racism card again? Why would two independent governing bodies group together to oppress a single country? Would citizens of the EU and US not want to go to one of the best schools regardless of locale?

newhouseb 4y ago

We also use GPT to perform actions in the software I build at work and we hit the same issue of inconsistency which lead me down a long rabbit hole to see if I could force an LLM to only emit grammatically correct output that follows a bespoke DSL (a DSL ideally safer and more precise than just eval'ing random AI-produced Python). I just finished writing up a long post [1] that describes how this can work on local models. It's a bit tricky to do via API efficiently, but hopefully OpenAI will give us the primitives one day to appropriately steer these models to do the right things (tm). [1] https://github.com/newhouseb/clownfish https://github.com/newhouseb/clownfish

elif 4y ago

I've been building a ChatGPT project this week and this explanation is so true it made me actually lol. Sometimes it's impressively flawless, sometimes it gets stuck in broken patterns... Even if the inputs are identical. "Monkey around with the prompt and pray" has replaced unit testing.

OOPMan 4y ago
Ooof
MilStdJunkie 4y ago
Exactly my experience. I've asked for some time metrics to see if the repeat is actually taking longer than the work.

csmpltn 4y ago

Why aren't they collecting feedback signals from people on the results produced by the model? "This was correct", "this was garbage", etc.

reportgunner 4y ago
What if someone was asking to generate garbage in Blender though?