5 ms·

How come general developer audiences aren't more acquainted with GPT-3 (and Codex in particular) capabilities? People in the twitter thread all seem completely

How come general developer audiences aren't more acquainted with GPT-3 (and Codex in particular) capabilities? People in the twitter thread all seem completely mind blown over an app that basically just passes your code to an existing API and prints the result. I don't want to sound negative of course, and I expect many of these apps coming up, until Codex stops being free (if they put it on the same pricing as text DaVinci model, which Codex is a fine-tuned version of, it will cost a ~cent per query). I'm just wondering how come the information about this type of app reaches most people way before the information about "the existence of Codex" reaches them. For all the publicity around Codex recently (and especially on HN), it still seems like the general IT audience is completely unaware of the (IMHO) most important thing going on in the field. And to anyone saying "all these examples are cherrypicked, Codex is stupid", I urge you to try Copilot and try to look at its output with the ~2019 perspective. I find it hard to beileve that anything but amazement is a proper reaction. And still, more people are aware of the recent BTC price, than this. Source: have been playing with Codex API for better part of every day for the last few weeks. Built an app that generates SQL for a custom schema, and have been using it in my daily work to boost ma productivity as a data scientis/engineer/analyst a lot.

vxNsr 4y ago
I was aware of and use copilot but I didn’t realize it was built on top of codex. And wasn’t even aware codex existed until you commmented. I read hn pretty regularly but unless you’re really excited about the AI space a lot of this news washes over you and you mostly ignore it.
- curl-up 4y ago
  This is what amazes me, since it seems like such big news, and people in the field are just not aware of it. Just for reference of what I am talking about, here is a piece of code that was generated, without any cherry picking at all (you just have to trust me on this, sorry) by allowing Codex to be aware of the database with some smart prompting (this is on a DB with music store data): Q: Best selling artist per country A: https://pastebin.com/qBVu2mvc https://pastebin.com/qBVu2mvc Needless to say, this query works and returns the data I wanted. Whether this is useful or not is up for discussion. But I cannot understand how it's not amazing.
  - benmmurphy 4y ago
    so it was smart enough to know that the total for the invoice could be safely used as the total sales for the artist because an invoice will only ever contain a single line item? no double counting because of multiple line items or misattribution because of different artists. that's a pretty crazy level of reasoning.
  - lupire 4y ago
    What happens if an Invoice has more than one Line? Looks like it will count the Invoice total for each line inatead of each likne's amount. If each invoice has 1 line, why does a line table exist?
    - curl-up 4y ago
      Ha, do I feel stupid now! Just checked, and seems like I misconfigured my system (the part that builds the Codex prompt from the schema info) - Invoice and InvoiceLine tabls got confused. After fixing it now, it works as expected (using IL.Total). Here are some more examples if someone's interested: https://pastebin.com/5Vr08N7z https://pastebin.com/5Vr08N7z
      - Izkata 4y ago
        This non-cherry-picked example is exactly one of the problems people are talking about though: It easily creates something that looks correct, and the users sees what they expect, so things get through that wouldn't have if they thought through the problem themselves.
        curl-up 4y ago
        Yeah, agree - building and verifying apps built on this models is very hard as they are hard to test extensively. But so are most other ML-based apps. The error in that example was on my side, not on the model. I mean, to me this seems simillar to saying that high-abstraction programming languages are a problem because their compilers are hard to develop and probably loaded with bugs.
P5fRxh5kUvp2th 4y ago
MS has been trying to get AI into intellisense for years now and I always turn it off. The lack of control over it just makes it annoying. In many ways it's faster to just type out the algorithm than it is to lay the algorithm out and spend the time trying to understand what's there so I can successfully convert the code to what I need. Then there's the lack of stability. Yesterday it did something different from what it's doing today, so I can't even use muscle memory to interact with it anymore. Intellisense has _always_ had that annoyance factor of getting in your way sometimes, forcing you to write code in a certain way to minimize that. All this just makes it more annoying and I don't believe anyone who claims it truly makes them more productive.
- curl-up 4y ago
  What I am talking about has nothing to do with Intellisense or your workflow. What I am saying is that, if someone in 2019 told you that there is a "thing" that is able to take a very complex sentence and qith high accuracy (and awareness of the database details) generate 50 lines of SQL, using CTEs, complex JOINs, subqueries, string formatting, date manipulation, etc, you would have been amazed. That thing now exists, and it didn't exist before. It is a complete phase shift and it cannot simply be viewed as a incremental improvement. This is a whole different beast. Using this beast as intellisense is just one application (called "Copilot") and it has all these annoyance factors sometimes. But I am not talking about that. To me, this is like we found a way to transform iron to gold with low energy usage, and people are complaining that gold is not that useful. And most chemists not even hearing about the news. I'm constantly amazed by this, every single day, as I read threads like this one.
  - P5fRxh5kUvp2th 4y ago
    I think starlink is a bigger deal by far.
    - curl-up 4y ago
      Can you elaborate? To me (but I know very little about it) it seems like part of the incremental progress in the internet availability. What am I missing?
iudqnolq 4y ago
I have no use for it, and don't expect ever having a use for it. 95% accurate, 99% accurate, and 99.9% accurate are all aweful in this context. It's something run repeatedly, so small chances will occur. Amoung it's failure states are being very, very wrong in ways that are hard for a skilled human to detect without more work that writing from scratch. And no one in the space is discussing ways to eliminate categories of bugs, only ways to reduce the frequency. Most of those solutions have the side effect of making the less frequent bugs harder to detect. On balance, that's worse. And, less importantly, it's only useful for writing boring code that should probably be generalized to an API. Sure, I write plenty of that, but it's not an exciting area to follow in my spare time.
- johnfn 4y ago
  > 95% accurate, 99% accurate, and 99.9% accurate are all aweful in this context. Really? What exactly is the bar then? I'd say most professionals I know hover somewhere between 95% and 99%.
  - lupire 4y ago
    What is the denominator? Lines of code? Functions?
  - iudqnolq 4y ago
    Assume it's per invocation, and each invocation generates a few dozen line function. How many such functions do you write when you get in the groove? If you multiply it out, you'll probably end up expecting a few bugs a day at 95%, similar to most humans might write. Except you're pretty used to the sorts of bugs you write, and the AI isn't you. So these bugs will be harder to find. So why is this better than writing by hand? Most of the hard work of programming is figuring out specs and debugging, not banging out well understood and specced implementations.
    - 8n4vidtmkvmk 4y ago
      wat? who's using the results without reading it and writing tests? not many bugs should be slipping by
kamban 4y ago
The problem is that they don't understand the practical ways it can be used. Even tech savvy people don’t yet get it. Even my CEO, a kind of technical person, did not understand the full potential until I explained some use cases. In one scenario, I took the slow running long MySQL query and rewrote that with Codex in 2 mins. But I think people have started to realize the potential now. Pitch: My app https://Elephas.app https://Elephas.app brings GPT-3 and Codex to all applications in MacOS. Many business professionals are using it.
FeepingCreature 4y ago
I'd love to try Codex if I could run it on a local GPU and finetune it for my own code. I'd even push to use it at work. But as we're writing in a niche language and our code is heavily problem domain dependent, I don't feel like making my workflow vulnerable to an external supplier, even aside the IP concerns. Call me when I can download and finetune the weights, like I can with Stable Diffusion.

vxNsr 4y ago

I was aware of and use copilot but I didn’t realize it was built on top of codex. And wasn’t even aware codex existed until you commmented. I read hn pretty regularly but unless you’re really excited about the AI space a lot of this news washes over you and you mostly ignore it.

curl-up 4y ago
This is what amazes me, since it seems like such big news, and people in the field are just not aware of it. Just for reference of what I am talking about, here is a piece of code that was generated, without any cherry picking at all (you just have to trust me on this, sorry) by allowing Codex to be aware of the database with some smart prompting (this is on a DB with music store data): Q: Best selling artist per country A: https://pastebin.com/qBVu2mvc https://pastebin.com/qBVu2mvc Needless to say, this query works and returns the data I wanted. Whether this is useful or not is up for discussion. But I cannot understand how it's not amazing.
- benmmurphy 4y ago
  so it was smart enough to know that the total for the invoice could be safely used as the total sales for the artist because an invoice will only ever contain a single line item? no double counting because of multiple line items or misattribution because of different artists. that's a pretty crazy level of reasoning.
- lupire 4y ago
  What happens if an Invoice has more than one Line? Looks like it will count the Invoice total for each line inatead of each likne's amount. If each invoice has 1 line, why does a line table exist?
  - curl-up 4y ago
    Ha, do I feel stupid now! Just checked, and seems like I misconfigured my system (the part that builds the Codex prompt from the schema info) - Invoice and InvoiceLine tabls got confused. After fixing it now, it works as expected (using IL.Total). Here are some more examples if someone's interested: https://pastebin.com/5Vr08N7z https://pastebin.com/5Vr08N7z
    - Izkata 4y ago
      This non-cherry-picked example is exactly one of the problems people are talking about though: It easily creates something that looks correct, and the users sees what they expect, so things get through that wouldn't have if they thought through the problem themselves.
      - curl-up 4y ago
        Yeah, agree - building and verifying apps built on this models is very hard as they are hard to test extensively. But so are most other ML-based apps. The error in that example was on my side, not on the model. I mean, to me this seems simillar to saying that high-abstraction programming languages are a problem because their compilers are hard to develop and probably loaded with bugs.

P5fRxh5kUvp2th 4y ago

MS has been trying to get AI into intellisense for years now and I always turn it off. The lack of control over it just makes it annoying. In many ways it's faster to just type out the algorithm than it is to lay the algorithm out and spend the time trying to understand what's there so I can successfully convert the code to what I need. Then there's the lack of stability. Yesterday it did something different from what it's doing today, so I can't even use muscle memory to interact with it anymore. Intellisense has _always_ had that annoyance factor of getting in your way sometimes, forcing you to write code in a certain way to minimize that. All this just makes it more annoying and I don't believe anyone who claims it truly makes them more productive.

curl-up 4y ago
What I am talking about has nothing to do with Intellisense or your workflow. What I am saying is that, if someone in 2019 told you that there is a "thing" that is able to take a very complex sentence and qith high accuracy (and awareness of the database details) generate 50 lines of SQL, using CTEs, complex JOINs, subqueries, string formatting, date manipulation, etc, you would have been amazed. That thing now exists, and it didn't exist before. It is a complete phase shift and it cannot simply be viewed as a incremental improvement. This is a whole different beast. Using this beast as intellisense is just one application (called "Copilot") and it has all these annoyance factors sometimes. But I am not talking about that. To me, this is like we found a way to transform iron to gold with low energy usage, and people are complaining that gold is not that useful. And most chemists not even hearing about the news. I'm constantly amazed by this, every single day, as I read threads like this one.
- P5fRxh5kUvp2th 4y ago
  I think starlink is a bigger deal by far.
  - curl-up 4y ago
    Can you elaborate? To me (but I know very little about it) it seems like part of the incremental progress in the internet availability. What am I missing?

iudqnolq 4y ago

I have no use for it, and don't expect ever having a use for it. 95% accurate, 99% accurate, and 99.9% accurate are all aweful in this context. It's something run repeatedly, so small chances will occur. Amoung it's failure states are being very, very wrong in ways that are hard for a skilled human to detect without more work that writing from scratch. And no one in the space is discussing ways to eliminate categories of bugs, only ways to reduce the frequency. Most of those solutions have the side effect of making the less frequent bugs harder to detect. On balance, that's worse. And, less importantly, it's only useful for writing boring code that should probably be generalized to an API. Sure, I write plenty of that, but it's not an exciting area to follow in my spare time.

johnfn 4y ago
> 95% accurate, 99% accurate, and 99.9% accurate are all aweful in this context. Really? What exactly is the bar then? I'd say most professionals I know hover somewhere between 95% and 99%.
- lupire 4y ago
  What is the denominator? Lines of code? Functions?
- iudqnolq 4y ago
  Assume it's per invocation, and each invocation generates a few dozen line function. How many such functions do you write when you get in the groove? If you multiply it out, you'll probably end up expecting a few bugs a day at 95%, similar to most humans might write. Except you're pretty used to the sorts of bugs you write, and the AI isn't you. So these bugs will be harder to find. So why is this better than writing by hand? Most of the hard work of programming is figuring out specs and debugging, not banging out well understood and specced implementations.
  - 8n4vidtmkvmk 4y ago
    wat? who's using the results without reading it and writing tests? not many bugs should be slipping by

kamban 4y ago

The problem is that they don't understand the practical ways it can be used. Even tech savvy people don’t yet get it. Even my CEO, a kind of technical person, did not understand the full potential until I explained some use cases. In one scenario, I took the slow running long MySQL query and rewrote that with Codex in 2 mins. But I think people have started to realize the potential now. Pitch: My app https://Elephas.app https://Elephas.app brings GPT-3 and Codex to all applications in MacOS. Many business professionals are using it.

FeepingCreature 4y ago

I'd love to try Codex if I could run it on a local GPU and finetune it for my own code. I'd even push to use it at work. But as we're writing in a niche language and our code is heavily problem domain dependent, I don't feel like making my workflow vulnerable to an external supplier, even aside the IP concerns. Call me when I can download and finetune the weights, like I can with Stable Diffusion.