3 ms·
We're considering doing something similar in my department. Based on some back-of-a-napkin math, it won't (yet) be cheap though, at about $0.40 per query. This
by vanviegen 3y ago
We're considering doing something similar in my department. Based on some back-of-a-napkin math, it won't (yet) be cheap though, at about $0.40 per query.
This is assuming the use of gpt-4 (chatgpt would be a lot cheaper, but it doesn't follow system level instructions very consistently, often spilling answers instead of replying with feedback, hints and leading questions).
Also, projected costs per query are high because we want to feed it quite a lot of input tokens. The assignment text, the student's code as well as the reference solution code.
- Paul-Craft 3y agoCould you possibly reduce those costs by fine-tuning the model with the assignment text and reference code, rather than including them in the prompt?
- vanviegen 3y agoOpenAI only offers finetuning for the pre-chatgpt models, so I doubt we'd get good enough results from that. Something like this might work with a self-hosted (llama) model though. We do have a (small) database of student work and teacher feedback for each assignment we could use for training (privacy permitting).
- nyberg 3y agoSo the course changes to a game of prompt injection for students aware of the state of LLMs
- vanviegen 3y agoThat's exactly what we're afraid of.