4 ms·

There are plenty of edge cases where it fails. However, the one thing that made me think it actually knows (for certain definitions of the word "knows") what it

There are plenty of edge cases where it fails. However, the one thing that made me think it actually knows (for certain definitions of the word "knows") what it's doing was asking it to re-write a non-trivial SQL query into equivalent relational algebra. I created a simplified schema from Northwind [0], gave it the CREATE TABLE statements for tables, and then some sort-of-TSV files for the values. It was able to not only produce reasonable outputs from various queries, but also to produce valid relational algebra for them. To me, that shows a fairly deep level of understanding of the underlying concepts. [0]: https://en.wikiversity.org/wiki/Database_Examples/Northwind https://en.wikiversity.org/wiki/Database_Examples/Northwind

lionkor 4y ago
Could you share the results?
- sgarland 4y ago
  Here's a gist [0]. You can see that it made a couple of minor semantic mistakes, which it corrected. It also regressed a bit at the end, but it _was_ able to correctly parse the (admittedly simple) math in a WHERE predicate and return a partially valid tuple, with respect to pricing. [0]: https://gist.github.com/stephanGarland/ed18f8f8fdc63a0b997f9df4aa88cc6e https://gist.github.com/stephanGarland/ed18f8f8fdc63a0b997f9...
jmull 4y ago
It's not just edge cases where it fails. It fails all the time at all kinds of things. I've been using chatgpt in my work, but I have to essentially know the answer it's going to give me because I have to catch all of its mistakes. It really, really nice for certain kinds of drudge work. Using northwind is probably not a good thing to use to evaluate chatgpt's general capability. It is very commonly used for examples of almost anything database-related, which means it's extremely well represented in chatgpt's training data. Chatgpt probably doesn't need to generalize or understand much of anything about northwind to answer your questions in terms of it. You need to try it on wacky problems specific to you.
- sgarland 4y ago
  > Using northwind is probably not a good thing to use to evaluate chatgpt's general capability. I did think of that, which is why I modified the schema, and removed any reference to Northwind (normally products are prefixed with "Northwind Traders"). That said, it isn't a particularly complex schema in my example, but it's a reasonable starting point for something a small business might use.

lionkor 4y ago

Could you share the results?

sgarland 4y ago
Here's a gist [0]. You can see that it made a couple of minor semantic mistakes, which it corrected. It also regressed a bit at the end, but it _was_ able to correctly parse the (admittedly simple) math in a WHERE predicate and return a partially valid tuple, with respect to pricing. [0]: https://gist.github.com/stephanGarland/ed18f8f8fdc63a0b997f9df4aa88cc6e https://gist.github.com/stephanGarland/ed18f8f8fdc63a0b997f9...

jmull 4y ago

It's not just edge cases where it fails. It fails all the time at all kinds of things. I've been using chatgpt in my work, but I have to essentially know the answer it's going to give me because I have to catch all of its mistakes. It really, really nice for certain kinds of drudge work. Using northwind is probably not a good thing to use to evaluate chatgpt's general capability. It is very commonly used for examples of almost anything database-related, which means it's extremely well represented in chatgpt's training data. Chatgpt probably doesn't need to generalize or understand much of anything about northwind to answer your questions in terms of it. You need to try it on wacky problems specific to you.

sgarland 4y ago
> Using northwind is probably not a good thing to use to evaluate chatgpt's general capability. I did think of that, which is why I modified the schema, and removed any reference to Northwind (normally products are prefixed with "Northwind Traders"). That said, it isn't a particularly complex schema in my example, but it's a reasonable starting point for something a small business might use.