5 ms·

Memorization is absolutely the most valuable part of GPT, for me. I can get natural language responses to documentation, basic scripting / sysadmin, and API qu

Memorization is absolutely the most valuable part of GPT, for me. I can get natural language responses to documentation, basic scripting / sysadmin, and API questions much more easily than searching other ways. While this is an academic interest point, and rightly tamps down on hype around replacing humans, it doesn't dissuade what I think are most peoples' basic use case: "I don't know or don't remember how to do X, can you show me?" This is finally a good enough "knowledge reference engine" that I can see being useful to those very people it is over hyped to replace.

vidarh 4y ago
And asking higher level questions than what you'd otherwise look up. E.g. I've had ChatGPT write forms, write API calls, put together skeletons for all kinds of things that I can easily verify and fix when it gets details from but that are time consuming to do manually. I've held back and been sceptical but I'm at the point where I'm preparing to integrate models all over the place because there are plenty of places where you can add sufficient checks that doing mostly ok much of the time is sufficient to already provide substantial time savings.
- zer00eyz 4y ago
  > I've held back and been sceptical but I'm at the point where I'm preparing to integrate models all over the place because there are plenty of places where you can add sufficient checks that doing mostly ok much of the time is sufficient to already provide substantial time savings. Im an old engineer. Simply put NO. If you don't understand it don't check it in. You are just getting code to cut and paste at a higher frequency and volume. At some point in time the fire will be burning around you and you won't have the tools to deal with it. Nothing about mostly, much and sufficient ever ends well when it has been done in the name of saving time.
  - vidarh 4y ago
    Nobody suggested checking in anything you don't understand. On the contrary. So maybe try reading again.
    - deleted 4y ago
      [deleted]
    - anon7725 4y ago
      The parent said: > I'm at the point where I'm preparing to integrate models all over the place Nobody understands these models right now. We don’t even have the weights. You may draw some artificial distinction between literally checking in the source code of a model into your git repo and making a call to some black box API that hosts it. And you may claim that doing so is no different than making a call to Twilio or whatever, but I think there is a major difference: nobody can make a claim about what an LLM will return or how it will return it, cannot make guarantees about how it will fail, etc. I agree with zer00eyz.
      - vidarh 4y ago
        I said that,and you're missing the point. We don't need to understand the models to be able to evaluate the output manually.
    - deleted 4y ago
      [deleted]
    - dahart 4y ago
      To be fair, “sufficient checks” and “mostly ok much of the time” does imply something not well understood to me. Maybe you could clarify instead of snapping at people, try writing again, if that’s not what you meant?
      - vidarh 4y ago
        For starter, "sufficient checks" does mean sufficient and that inherently means I need to fully understabd the risks. You're jumping to conclusions not supported by the comment at all. Also, the comment has two parts: One about writing code, and one about integrating models in workflows. To the latter, the point is that for a whole lot of uses you can trivially ensure the failure modes are safe. E.g. I am integrating gpt with my email. "Mostly ok most of the time" applies to things like e.g. summaries and prioritisation, because worst case I just get to an email a bit later. "Sufficient checks" applies to things like writing proposed replies: There's no way I'd send one without reading it, and it's sufficient for me to read through it before pressing send (and making adjustments as needed). Failures here would matter if I intended to make a product of it, but as a productivity tool for myself it just needs to be close enough. There are a whole lot of possibilities like that. But even for coding related tasks there are a whole lot of low risk tasks,such as e.g. generating HTML or CSS, or provide usage examples, or providing a scaffold for something you know well how to do but which are time consuming. If you're trying to make it do things that'd be time consuming to verify sufficiently well, then that's a bad use. The good uses are those where errors are low impact and easy to catch.
  - simonw 4y ago
    "You are just getting code to cut and paste at a higher frequency and volume" genuinely sounds like the key value proposition of ChatGPT for coding to me. I treat its output like I would treat a PR from a brand new apprentice engineer on my team: review it carefully, provide some feedback and iterate a few times, accept with tests.
    - vidarh 4y ago
      Exactly. And my point in the first place was that it's most useful for those kinds of tasks you might hand to an apprentice where the apprentice might go away, spend a lot of time doing research and distill it down to some code that is simple, likely not all that great, but saves me time. E.g. some tasks I've used it for recently: * Giving me an outline of a JMAP client so I can pull down stuff from my e-mail to feed to GPT. * Giving me an outline of an OpenAPI client. * Giving me an index page and a layout for a website, including a simple starting point for the CSS that did a reset and added basic styling for the nav bar, forms and "hero "sections. * Giving me an outline of a Stripe API integration. * Writing a simple DNS server. * Writing a simple web server capable of running Sinatra apps via Rack. None of these were complex code that'd hide obscure bugs. None were big chunks of code. All of them were simple code that was always going to have big, gaping holes and sub-optimal choices that'd need to be addressed, but that was fine because they were scaffolding that saved me starting from scratch (and the last two were not intended to turn into anything, but just exploring what it could do) That's where the biggest savings are for me, because if I asked it to generate particularly complex stuff, I'd end up spending ages getting comfortable it'd done it right and verifying it. But the simple but tedious stuff is something it's great for.
  - Karunamon 4y ago
    Nobody said one word about checking in something they don't understand. That applies to copying from stackoverflow as much as it does from an LLM or copilot.
  - poniko 4y ago
    Isn't that what we all have been doing with google/stackoverflow .. how do I solve xx? Aha seems right, copy, paste and a quick format.. cross fingers and run.
  - pixl97 4y ago
    >If you don't understand it don't check it in. I work in code security, and after helping any number of customers, I can tell you this isn't how far too many programmers work. A client recently had a problem with a project that had over 1200 node_modules. 1200... Let that sink in. There is absolutely no way in hell they even had any idea about a small portion of the code they were including.
    - com2kid 4y ago
      > A client recently had a problem with a project that had over 1200 node_modules. # of Node modules is such a useless metric. In any given project, a large # of node modules are part of the test, build, and linting frameworks. If I go to C++ land and count the number of #import statements, it wouldn't tell me anything. How many classes do large Java projects use? Typically some absurd number.
    - teaearlgraycold 4y ago
      Are those direct dependencies or the full dependency tree?
  - hn_throwaway_99 4y ago
    I think you are misunderstanding. The post you are replying to clearly said they were reviewing output code before checking it in. The fact that we don't understand how the models work is irrelevant (we don't understand how the human brain works, either) - all we need to understand is how the output works. I had a conversation with ChatGPT where I asked it to write me a piece of code. After it wrote the code, I reviewed it, and I told ChatGPT that it had a subtle bug. ChatGPT then fixed the bug itself, and wrote an English description about how the fix it added would prevent the bug.
  - solarkraft 4y ago
    They say they verify the code, do they should understand it. But also: Have you heard of StackOverflow? Copy/pasting code you don't (fully) understand is already a common practice that seems to mostly work well.
User23 4y ago
The main problem with using GPT-3 (and maybe 4 I dunno) in that way is it will happily bullshit you to the point of making up fake references. For example it quoted me "Section 6.2" of the Go Programming Language Specification to support its answer when I asked it how a particular conversion is specified.
- dpkirchner 4y ago
  Do you remember the prompt (it should be in the history column)? I'm curious how it came up with the section numbering.
- billythemaniam 4y ago
  So far GPT-4 seems to improve on this problem. Still happens but less frequent.
  - mtlmtlmtlmtl 4y ago
    I'm assuming you base this on hard empirical data and not just that it feels like it when you use it? ;)
rubendv 4y ago
I guess it is in OpenAI's best interest to downplay the memorization aspect in favor of the logical reasoning angle. If it turns out that GPT is memorizing and reproducing copyrighted data, it could land them in legal trouble.
- brookst 4y ago
  On the bright side it would mean they have invented an amazing compression algorithm, given the model size and amount of text it can produce.
- dragonwriter 4y ago
  > I guess it is in OpenAI’s best interest to downplay the memorization aspect in favor of the logical reasoning angle. If it turns out that GPT is memorizing and reproducing copyrighted data, it could land them in legal trouble. Its also in their interest, since it encourages people to attempt to build businesses on top of its “logical reasoning” capacities. And as long as its within the realm of spin rather than direct false statements, it probably avoids creating them legal problems for fraud, although the difference in effect is…nonexistent.
petesergeant 4y ago
I dunno, I use ChatGPT for exactly the same thing as you, and people are always quite surprised when I say that's its main value to me, so I think people have very different ideas of what it excels at
soco 4y ago
Good luck getting ChatGPT to explain a cron expression like "0 30 5 * * 3". I mean, it will explain, but mixing up everything. How many other mistakes it might make?
- messe 4y ago
  Isn't that an invalid cron expression? It has six fields instead of five, and says to run at the 30th hour of the day.
  - soco 4y ago
    It's a valid Quartz scheduler task where it starts with the seconds. But right, I shouldn't have asked ChatGPT about cron when it's actually quartz - and indeed with the changed prompt it will describe it correctly. Edit: actually almost correctly: " - '3' represents the day of the week on which the task will be executed. In this case, the value is 3, which represents Wednesday (Sunday is 1, Monday is 2, and so on)."
    - jldugger 4y ago
      Lol, GPT exposing bugs in the wetware
      - soco 4y ago
        Actually, no. It gave a wrong answer in full confidence then HN analyzed it to expose the bug(s).
- nielsole 4y ago
  [...] > So the cron expression `0 30 5 * * 3` means "run the cron job every Wednesday at 5:30 AM". It explains the five Cron parameters but the doesn't pick up that six were provided. Oh well
  - messe 4y ago
    I got a similar but incorrect result from ChatGPT: "So, the expression "0 30 5 * * 3" means that the command should be executed at 5:30 AM on the 5th day of every month, but only if that day is a Wednesday" However, the crontab is invalid. It has five fields instead of six, and the 30 corresponds to the hour field. "30 5 * * 3" without the leading zero, would correspond to run every Wednesday at 5:30 AM though. I suspect the fact that it has six fields instead of five is confusing it, and it's interpreting the 5 as a day of the month and as the hour
surrTurr 4y ago
I recently built something related (QA system for Zotero powered by LangChain & GPT). Works really well. https://twitter.com/alexweichart/status/1637211755049897985?s=46&t=D9j6hLVSKTKnAbXYHtIrFg https://twitter.com/alexweichart/status/1637211755049897985?...

vidarh 4y ago

And asking higher level questions than what you'd otherwise look up. E.g. I've had ChatGPT write forms, write API calls, put together skeletons for all kinds of things that I can easily verify and fix when it gets details from but that are time consuming to do manually. I've held back and been sceptical but I'm at the point where I'm preparing to integrate models all over the place because there are plenty of places where you can add sufficient checks that doing mostly ok much of the time is sufficient to already provide substantial time savings.

zer00eyz 4y ago
> I've held back and been sceptical but I'm at the point where I'm preparing to integrate models all over the place because there are plenty of places where you can add sufficient checks that doing mostly ok much of the time is sufficient to already provide substantial time savings. Im an old engineer. Simply put NO. If you don't understand it don't check it in. You are just getting code to cut and paste at a higher frequency and volume. At some point in time the fire will be burning around you and you won't have the tools to deal with it. Nothing about mostly, much and sufficient ever ends well when it has been done in the name of saving time.
- vidarh 4y ago
  Nobody suggested checking in anything you don't understand. On the contrary. So maybe try reading again.
  - deleted 4y ago
    [deleted]
  - anon7725 4y ago
    The parent said: > I'm at the point where I'm preparing to integrate models all over the place Nobody understands these models right now. We don’t even have the weights. You may draw some artificial distinction between literally checking in the source code of a model into your git repo and making a call to some black box API that hosts it. And you may claim that doing so is no different than making a call to Twilio or whatever, but I think there is a major difference: nobody can make a claim about what an LLM will return or how it will return it, cannot make guarantees about how it will fail, etc. I agree with zer00eyz.
    - vidarh 4y ago
      I said that,and you're missing the point. We don't need to understand the models to be able to evaluate the output manually.
  - deleted 4y ago
    [deleted]
  - dahart 4y ago
    To be fair, “sufficient checks” and “mostly ok much of the time” does imply something not well understood to me. Maybe you could clarify instead of snapping at people, try writing again, if that’s not what you meant?
    - vidarh 4y ago
      For starter, "sufficient checks" does mean sufficient and that inherently means I need to fully understabd the risks. You're jumping to conclusions not supported by the comment at all. Also, the comment has two parts: One about writing code, and one about integrating models in workflows. To the latter, the point is that for a whole lot of uses you can trivially ensure the failure modes are safe. E.g. I am integrating gpt with my email. "Mostly ok most of the time" applies to things like e.g. summaries and prioritisation, because worst case I just get to an email a bit later. "Sufficient checks" applies to things like writing proposed replies: There's no way I'd send one without reading it, and it's sufficient for me to read through it before pressing send (and making adjustments as needed). Failures here would matter if I intended to make a product of it, but as a productivity tool for myself it just needs to be close enough. There are a whole lot of possibilities like that. But even for coding related tasks there are a whole lot of low risk tasks,such as e.g. generating HTML or CSS, or provide usage examples, or providing a scaffold for something you know well how to do but which are time consuming. If you're trying to make it do things that'd be time consuming to verify sufficiently well, then that's a bad use. The good uses are those where errors are low impact and easy to catch.
- simonw 4y ago
  "You are just getting code to cut and paste at a higher frequency and volume" genuinely sounds like the key value proposition of ChatGPT for coding to me. I treat its output like I would treat a PR from a brand new apprentice engineer on my team: review it carefully, provide some feedback and iterate a few times, accept with tests.
  - vidarh 4y ago
    Exactly. And my point in the first place was that it's most useful for those kinds of tasks you might hand to an apprentice where the apprentice might go away, spend a lot of time doing research and distill it down to some code that is simple, likely not all that great, but saves me time. E.g. some tasks I've used it for recently: * Giving me an outline of a JMAP client so I can pull down stuff from my e-mail to feed to GPT. * Giving me an outline of an OpenAPI client. * Giving me an index page and a layout for a website, including a simple starting point for the CSS that did a reset and added basic styling for the nav bar, forms and "hero "sections. * Giving me an outline of a Stripe API integration. * Writing a simple DNS server. * Writing a simple web server capable of running Sinatra apps via Rack. None of these were complex code that'd hide obscure bugs. None were big chunks of code. All of them were simple code that was always going to have big, gaping holes and sub-optimal choices that'd need to be addressed, but that was fine because they were scaffolding that saved me starting from scratch (and the last two were not intended to turn into anything, but just exploring what it could do) That's where the biggest savings are for me, because if I asked it to generate particularly complex stuff, I'd end up spending ages getting comfortable it'd done it right and verifying it. But the simple but tedious stuff is something it's great for.
- Karunamon 4y ago
  Nobody said one word about checking in something they don't understand. That applies to copying from stackoverflow as much as it does from an LLM or copilot.
- poniko 4y ago
  Isn't that what we all have been doing with google/stackoverflow .. how do I solve xx? Aha seems right, copy, paste and a quick format.. cross fingers and run.
- pixl97 4y ago
  >If you don't understand it don't check it in. I work in code security, and after helping any number of customers, I can tell you this isn't how far too many programmers work. A client recently had a problem with a project that had over 1200 node_modules. 1200... Let that sink in. There is absolutely no way in hell they even had any idea about a small portion of the code they were including.
  - com2kid 4y ago
    > A client recently had a problem with a project that had over 1200 node_modules. # of Node modules is such a useless metric. In any given project, a large # of node modules are part of the test, build, and linting frameworks. If I go to C++ land and count the number of #import statements, it wouldn't tell me anything. How many classes do large Java projects use? Typically some absurd number.
  - teaearlgraycold 4y ago
    Are those direct dependencies or the full dependency tree?
- hn_throwaway_99 4y ago
  I think you are misunderstanding. The post you are replying to clearly said they were reviewing output code before checking it in. The fact that we don't understand how the models work is irrelevant (we don't understand how the human brain works, either) - all we need to understand is how the output works. I had a conversation with ChatGPT where I asked it to write me a piece of code. After it wrote the code, I reviewed it, and I told ChatGPT that it had a subtle bug. ChatGPT then fixed the bug itself, and wrote an English description about how the fix it added would prevent the bug.
- solarkraft 4y ago
  They say they verify the code, do they should understand it. But also: Have you heard of StackOverflow? Copy/pasting code you don't (fully) understand is already a common practice that seems to mostly work well.

User23 4y ago

The main problem with using GPT-3 (and maybe 4 I dunno) in that way is it will happily bullshit you to the point of making up fake references. For example it quoted me "Section 6.2" of the Go Programming Language Specification to support its answer when I asked it how a particular conversion is specified.

dpkirchner 4y ago
Do you remember the prompt (it should be in the history column)? I'm curious how it came up with the section numbering.
billythemaniam 4y ago
So far GPT-4 seems to improve on this problem. Still happens but less frequent.
- mtlmtlmtlmtl 4y ago
  I'm assuming you base this on hard empirical data and not just that it feels like it when you use it? ;)

rubendv 4y ago

I guess it is in OpenAI's best interest to downplay the memorization aspect in favor of the logical reasoning angle. If it turns out that GPT is memorizing and reproducing copyrighted data, it could land them in legal trouble.

brookst 4y ago
On the bright side it would mean they have invented an amazing compression algorithm, given the model size and amount of text it can produce.
dragonwriter 4y ago
> I guess it is in OpenAI’s best interest to downplay the memorization aspect in favor of the logical reasoning angle. If it turns out that GPT is memorizing and reproducing copyrighted data, it could land them in legal trouble. Its also in their interest, since it encourages people to attempt to build businesses on top of its “logical reasoning” capacities. And as long as its within the realm of spin rather than direct false statements, it probably avoids creating them legal problems for fraud, although the difference in effect is…nonexistent.

petesergeant 4y ago

I dunno, I use ChatGPT for exactly the same thing as you, and people are always quite surprised when I say that's its main value to me, so I think people have very different ideas of what it excels at

soco 4y ago

Good luck getting ChatGPT to explain a cron expression like "0 30 5 * * 3". I mean, it will explain, but mixing up everything. How many other mistakes it might make?

messe 4y ago
Isn't that an invalid cron expression? It has six fields instead of five, and says to run at the 30th hour of the day.
- soco 4y ago
  It's a valid Quartz scheduler task where it starts with the seconds. But right, I shouldn't have asked ChatGPT about cron when it's actually quartz - and indeed with the changed prompt it will describe it correctly. Edit: actually almost correctly: " - '3' represents the day of the week on which the task will be executed. In this case, the value is 3, which represents Wednesday (Sunday is 1, Monday is 2, and so on)."
  - jldugger 4y ago
    Lol, GPT exposing bugs in the wetware
    - soco 4y ago
      Actually, no. It gave a wrong answer in full confidence then HN analyzed it to expose the bug(s).
nielsole 4y ago
[...] > So the cron expression `0 30 5 * * 3` means "run the cron job every Wednesday at 5:30 AM". It explains the five Cron parameters but the doesn't pick up that six were provided. Oh well
- messe 4y ago
  I got a similar but incorrect result from ChatGPT: "So, the expression "0 30 5 * * 3" means that the command should be executed at 5:30 AM on the 5th day of every month, but only if that day is a Wednesday" However, the crontab is invalid. It has five fields instead of six, and the 30 corresponds to the hour field. "30 5 * * 3" without the leading zero, would correspond to run every Wednesday at 5:30 AM though. I suspect the fact that it has six fields instead of five is confusing it, and it's interpreting the 5 as a day of the month and as the hour

surrTurr 4y ago

I recently built something related (QA system for Zotero powered by LangChain & GPT). Works really well. https://twitter.com/alexweichart/status/1637211755049897985?s=46&t=D9j6hLVSKTKnAbXYHtIrFg https://twitter.com/alexweichart/status/1637211755049897985?...