5 ms·

This is what amazes me, since it seems like such big news, and people in the field are just not aware of it. Just for reference of what I am talking about, here

This is what amazes me, since it seems like such big news, and people in the field are just not aware of it. Just for reference of what I am talking about, here is a piece of code that was generated, without any cherry picking at all (you just have to trust me on this, sorry) by allowing Codex to be aware of the database with some smart prompting (this is on a DB with music store data): Q: Best selling artist per country A: https://pastebin.com/qBVu2mvc https://pastebin.com/qBVu2mvc Needless to say, this query works and returns the data I wanted. Whether this is useful or not is up for discussion. But I cannot understand how it's not amazing.

benmmurphy 4y ago
so it was smart enough to know that the total for the invoice could be safely used as the total sales for the artist because an invoice will only ever contain a single line item? no double counting because of multiple line items or misattribution because of different artists. that's a pretty crazy level of reasoning.
lupire 4y ago
What happens if an Invoice has more than one Line? Looks like it will count the Invoice total for each line inatead of each likne's amount. If each invoice has 1 line, why does a line table exist?
- curl-up 4y ago
  Ha, do I feel stupid now! Just checked, and seems like I misconfigured my system (the part that builds the Codex prompt from the schema info) - Invoice and InvoiceLine tabls got confused. After fixing it now, it works as expected (using IL.Total). Here are some more examples if someone's interested: https://pastebin.com/5Vr08N7z https://pastebin.com/5Vr08N7z
  - Izkata 4y ago
    This non-cherry-picked example is exactly one of the problems people are talking about though: It easily creates something that looks correct, and the users sees what they expect, so things get through that wouldn't have if they thought through the problem themselves.
    - curl-up 4y ago
      Yeah, agree - building and verifying apps built on this models is very hard as they are hard to test extensively. But so are most other ML-based apps. The error in that example was on my side, not on the model. I mean, to me this seems simillar to saying that high-abstraction programming languages are a problem because their compilers are hard to develop and probably loaded with bugs.

benmmurphy 4y ago

so it was smart enough to know that the total for the invoice could be safely used as the total sales for the artist because an invoice will only ever contain a single line item? no double counting because of multiple line items or misattribution because of different artists. that's a pretty crazy level of reasoning.

lupire 4y ago

What happens if an Invoice has more than one Line? Looks like it will count the Invoice total for each line inatead of each likne's amount. If each invoice has 1 line, why does a line table exist?

curl-up 4y ago
Ha, do I feel stupid now! Just checked, and seems like I misconfigured my system (the part that builds the Codex prompt from the schema info) - Invoice and InvoiceLine tabls got confused. After fixing it now, it works as expected (using IL.Total). Here are some more examples if someone's interested: https://pastebin.com/5Vr08N7z https://pastebin.com/5Vr08N7z
- Izkata 4y ago
  This non-cherry-picked example is exactly one of the problems people are talking about though: It easily creates something that looks correct, and the users sees what they expect, so things get through that wouldn't have if they thought through the problem themselves.
  - curl-up 4y ago
    Yeah, agree - building and verifying apps built on this models is very hard as they are hard to test extensively. But so are most other ML-based apps. The error in that example was on my side, not on the model. I mean, to me this seems simillar to saying that high-abstraction programming languages are a problem because their compilers are hard to develop and probably loaded with bugs.