3 ms·

What evals did you use to compare that to using an existing language with a large corpus of training data?

• ModernMech 2mo ago
  I've got two experiments running now and I'm going to do a third soon. The first area is linear algebra, where there's a readily available notation for the AI to operationalize. In this area it's very easy for the AI to one-shot write correct algorithms that are shorter than typical languages because they are already written down, and the notation is very compact, so all it has to do is a direct translation from a textbook. I'm now working on comparing with algorithms it doesn't already know. The second area I'm working on now doesn't have a readily available notation, which is state machines. Here, the AI can concoct a very terse state machine representation and write very complex state machines that can be statically analyzed so it has a better time than writing in a plain language without that capability. Now I'm trying to test how it fares against other state machine DSLs. The next area I will move to after this I think is music, which also has a readily available notation that AI can operationalize. No numbers to report yet but I'll publish my research when it's done.