6 ms·

Running Python micro-benchmarks using the ChatGPT Code Interpreter alpha

65 points by srijan4 4y ago

tomviner 4y ago
I wonder what's the simplest Langchain agent that can recreate this?
simonw 4y ago
This entire project took three prompts: In Python create an in-memory SQLite database with 100 tables each with 10 columns. Time how long it takes to execute PRAGMA schema_version against that database 100 times. Then add another 100 tables and time PRAGMA schema_version 100 times again. Now run the whole benchmark a second time, but instead of PRAGMA schema_version time how long it takes to run hashlib.md5(db.execute( "select group_concat(sql) from sqlite_master" ).fetchall()[0]).hexdigest() instead Then: Plot those benchmark results over time on a chart Then: Run the benchmark again in order to draw a better chart with measurements for every point between 1 and 200 tables
btilly 4y ago
OK, now I have developed sympathy for people who are color-blind. Which line is which???
- srijan4 4y ago
  That was my first thought as well. I'm sure a 4th prompt to choose different colors for the graphs would work.
- simonw 4y ago
  Hah, good call. I'll tell it to do that better.
  - simonw 4y ago
    Prompt: > One more chart plot, this time with colors that differ more (and are OK for people who are color blind) https://static.simonwillison.net/static/2023/better-colors.png https://static.simonwillison.net/static/2023/better-colors.p... It picked magenta and dark green.
- fellerts 4y ago
  I'm red-green blind but had no issue telling the plot lines apart, because the two lines have different contrast. Try converting the image to grayscale and you'll see that they are different. I'm genuinely curious: why did you have trouble telling them apart?
  - herewego 4y ago
    Depends on the person. I have trouble with red and green traffic lights, despite what others have indicated are clear contrast/brightness differences. I use the position of the lights and Tesla’s green light detector to combat this. Aren’t eyes weird?
  - btilly 4y ago
    I'm over 50, and have gotten used to using color as a way to distinguish rather than exact shades. So I'm missing the cue I naturally relay on, and my vision is sufficiently blurry to make the one that was available hard to use.
anonzzzies 4y ago
I have access to plug-ins as well, but I have some reading comprehension issues; how do I use the ones by openai themselves like this one? The other ones are in the chatgpt plug-in store but the interpreter and browser ones are not?
- simonw 4y ago
  The code interpreter and browse mode things don't work like other plugins. You can't install them alongside plugins (probably for the best given prompt injection risks) - you can pick one of "Plugins" mode or "Browse" mode or "Code Interpreter" mode.
- kolinko 4y ago
  Not everyone has access to these ones yet
  - anonzzzies 4y ago
    Ah that would explain it; thought the plug-in access would mean all they have. So it would need to appear in the list with the rest?
cjbprime 4y ago
> And of course, they have zero understanding of anything at all. They’re a next-token-predicting machine with an unimaginably large training set. Oof, this fallacy is so frustrating. Being a next-token-predicting machine doesn't imply "having no understanding of anything". The best way to correctly predict the next token in the answer to a question is to understand the question properly, and reason your way to that true answer. So that's what it tries to do (with inconsistent success). Seeing the tool this way should also remove the disbelief that it works. It works because it starts to model not only the surface statistics, but the causes of those statistics, in order to make predictions that are less wrong. That's how it gets a world model from a bunch of sample text.
- adammarples 4y ago
  Yes the best way to predict the next token would be to be a real AGI with an accurate world model. That's not what it is though, it's a very good language model.
cleanchit 4y ago
I can't be the only software developer who's feeling scared.
- PhilipRoman 4y ago
  For the CRUD app writers - yeah this is bad news. But it was always clear that it was a bubble. If you're actually solving new problems, it works more like code completion. Unix was written in assembly with the Ed editor. Even Notepad and any modern language probably represents a 100x increase in productivity compared to that - much more than ChatGPT can do. The field of software engineering will only grow from this. From what I've seen - most places have way more work than available programmers. Jira tickets stay unsolved for years. Maybe with the increased productivity we can actually clear our backlogs one day.
  - Szpadel 4y ago
    Not sure about you, but I also have a huge backlog of projects that don't exist yet and I would like to create them. but usually they're too complex to do over the weekend and I don't have that much time to spare. I already tried to approach one of those projects but I quickly failed as gpt knowledge was too outdated for the library that I needed to use and I didn't figure out how to "patch" gpt knowledge. in this case I'm not very familiar with the stack I needed for that project ( and I think future versions of LLM could get better at bridging that gap) but for tasks that I'm more familiar with I noticed a significant increase in productivity
- bugglebeetle 4y ago
  I hope there’s still room for improvement. I haven’t been able to get ChatGPT to do anything coherent for stuff with > 5 functions or logic that is even the slightest complex. It likewise frequently stumbles/hallucinates when trying to integrate revisions. This is all using GPT-4 as well. What’s it’s absolutely fantastic at is the small, one-off scripts that were tedious and time-consuming to write, integrating well-specified changes into existing code, tests, and boiler plate config stuff. I know this is all “for now” talk, but it will be interesting to see how quickly or if at all it can get to production-ready code in the absence of pretty thorough review by someone who knows how stuff actually works. Natural language is a fantastic new interface, but (for now) you still have to know how to describe stuff that actually works.
sheepscreek 4y ago
> And of course, they have zero understanding of anything at all. They’re a next-token-predicting machine with an unimaginably large training set. How do the neurons in our brain “understand” anything? Can you believe our body is made up of > 100 trillion independent cells - each one is a “living” thing in its own right. We are not “one” living thing, but a collection of many living things - working together to achieve unbelievable results. In that regard, we are more similar than anything ever done in computer science resembling our minds. LLMs work on probabilities and our neurons work on analog signals - both are across a range and not binary. …to be continued.
- KyeRussell 4y ago
  I am honestly a bit disappointed that Simon has spent so much time playing with LLMs, and does have some interesting findings to share, but feels the need to include these inferences which I’d usually expect from drive-by HN skeptics.
- imtringued 4y ago
  Our brains don't do backpropagation.

65 points by srijan4 4y ago

I wonder what's the simplest Langchain agent that can recreate this?

This entire project took three prompts: In Python create an in-memory SQLite database with 100 tables each with 10 columns. Time how long it takes to execute PRAGMA schema_version against that database 100 times. Then add another 100 tables and time PRAGMA schema_version 100 times again. Now run the whole benchmark a second time, but instead of PRAGMA schema_version time how long it takes to run hashlib.md5(db.execute( "select group_concat(sql) from sqlite_master" ).fetchall()[0]).hexdigest() instead Then: Plot those benchmark results over time on a chart Then: Run the benchmark again in order to draw a better chart with measurements for every point between 1 and 200 tables

btilly 4y ago

OK, now I have developed sympathy for people who are color-blind. Which line is which???

srijan4 4y ago
That was my first thought as well. I'm sure a 4th prompt to choose different colors for the graphs would work.
simonw 4y ago
Hah, good call. I'll tell it to do that better.
- simonw 4y ago
  Prompt: > One more chart plot, this time with colors that differ more (and are OK for people who are color blind) https://static.simonwillison.net/static/2023/better-colors.png https://static.simonwillison.net/static/2023/better-colors.p... It picked magenta and dark green.
fellerts 4y ago
I'm red-green blind but had no issue telling the plot lines apart, because the two lines have different contrast. Try converting the image to grayscale and you'll see that they are different. I'm genuinely curious: why did you have trouble telling them apart?
- herewego 4y ago
  Depends on the person. I have trouble with red and green traffic lights, despite what others have indicated are clear contrast/brightness differences. I use the position of the lights and Tesla’s green light detector to combat this. Aren’t eyes weird?
- btilly 4y ago
  I'm over 50, and have gotten used to using color as a way to distinguish rather than exact shades. So I'm missing the cue I naturally relay on, and my vision is sufficiently blurry to make the one that was available hard to use.

anonzzzies 4y ago

I have access to plug-ins as well, but I have some reading comprehension issues; how do I use the ones by openai themselves like this one? The other ones are in the chatgpt plug-in store but the interpreter and browser ones are not?

simonw 4y ago
The code interpreter and browse mode things don't work like other plugins. You can't install them alongside plugins (probably for the best given prompt injection risks) - you can pick one of "Plugins" mode or "Browse" mode or "Code Interpreter" mode.
kolinko 4y ago
Not everyone has access to these ones yet
- anonzzzies 4y ago
  Ah that would explain it; thought the plug-in access would mean all they have. So it would need to appear in the list with the rest?

cjbprime 4y ago

> And of course, they have zero understanding of anything at all. They’re a next-token-predicting machine with an unimaginably large training set. Oof, this fallacy is so frustrating. Being a next-token-predicting machine doesn't imply "having no understanding of anything". The best way to correctly predict the next token in the answer to a question is to understand the question properly, and reason your way to that true answer. So that's what it tries to do (with inconsistent success). Seeing the tool this way should also remove the disbelief that it works. It works because it starts to model not only the surface statistics, but the causes of those statistics, in order to make predictions that are less wrong. That's how it gets a world model from a bunch of sample text.

adammarples 4y ago
Yes the best way to predict the next token would be to be a real AGI with an accurate world model. That's not what it is though, it's a very good language model.

cleanchit 4y ago

I can't be the only software developer who's feeling scared.

PhilipRoman 4y ago
For the CRUD app writers - yeah this is bad news. But it was always clear that it was a bubble. If you're actually solving new problems, it works more like code completion. Unix was written in assembly with the Ed editor. Even Notepad and any modern language probably represents a 100x increase in productivity compared to that - much more than ChatGPT can do. The field of software engineering will only grow from this. From what I've seen - most places have way more work than available programmers. Jira tickets stay unsolved for years. Maybe with the increased productivity we can actually clear our backlogs one day.
- Szpadel 4y ago
  Not sure about you, but I also have a huge backlog of projects that don't exist yet and I would like to create them. but usually they're too complex to do over the weekend and I don't have that much time to spare. I already tried to approach one of those projects but I quickly failed as gpt knowledge was too outdated for the library that I needed to use and I didn't figure out how to "patch" gpt knowledge. in this case I'm not very familiar with the stack I needed for that project ( and I think future versions of LLM could get better at bridging that gap) but for tasks that I'm more familiar with I noticed a significant increase in productivity
bugglebeetle 4y ago
I hope there’s still room for improvement. I haven’t been able to get ChatGPT to do anything coherent for stuff with > 5 functions or logic that is even the slightest complex. It likewise frequently stumbles/hallucinates when trying to integrate revisions. This is all using GPT-4 as well. What’s it’s absolutely fantastic at is the small, one-off scripts that were tedious and time-consuming to write, integrating well-specified changes into existing code, tests, and boiler plate config stuff. I know this is all “for now” talk, but it will be interesting to see how quickly or if at all it can get to production-ready code in the absence of pretty thorough review by someone who knows how stuff actually works. Natural language is a fantastic new interface, but (for now) you still have to know how to describe stuff that actually works.

sheepscreek 4y ago

> And of course, they have zero understanding of anything at all. They’re a next-token-predicting machine with an unimaginably large training set. How do the neurons in our brain “understand” anything? Can you believe our body is made up of > 100 trillion independent cells - each one is a “living” thing in its own right. We are not “one” living thing, but a collection of many living things - working together to achieve unbelievable results. In that regard, we are more similar than anything ever done in computer science resembling our minds. LLMs work on probabilities and our neurons work on analog signals - both are across a range and not binary. …to be continued.

KyeRussell 4y ago
I am honestly a bit disappointed that Simon has spent so much time playing with LLMs, and does have some interesting findings to share, but feels the need to include these inferences which I’d usually expect from drive-by HN skeptics.
imtringued 4y ago
Our brains don't do backpropagation.