I was taking a political compass quiz the other day and I was frustrated that I couldn't quite express myself through the multiple choice answers. I have some

I was taking a political compass quiz the other day and I was frustrated that I couldn't quite express myself through the multiple choice answers. I have some political beliefs that are hard to express with just "Agree/disagree" sentiments. It occurred to me while I was trying to explain myself to the multiple choice form that you could use ChatGPT, or similar LLM, for this. My original goal was "interactive questions" - so a question could ask "Do you think a wealth tax is a good idea" and the user could say "What kind of wealth tax, who would it apply to" and so on to refine the question to something they were comfortable answering with a yes/no. Experimenting with this I discovered that it would actually take a lot of tokens (and therefore money) to have real conversations about the questions. Instead, I changed my vision from "interactive questions" to "short essay responses". Basically, you get a prompt for a political question, write your thoughts about that prompt, and then ChatGPT (3.5 - again, money) grades your essays on two dimensions according to a couple rubrics I wrote for it. I have shown this to a few people and taken the test a couple times myself. I think it's working reasonably well. At the end of the test you can see how the LLM graded each of your essays and you can click "thumbs up" or "thumbs down" if you agree or disagree with the LLM. So far, I have 44 bits of feedback 5 of which are negative, which suggests to me that, for the most part, the LLM does a good job at grading. One problem I have already noticed with the concept is that people often write answers that are much too short - e.g. some people will just respond by typing out "Yes", "No", "False" or similar one or two word replies. Such replies kind of defeat the purpose of the essay writing aspect and the LLM does have trouble fully grading them. I'm thinking about requiring a minimum number of words, or having the LLM prompt the user "Why do you think X?". If I keep working on this one thing I'd like to do is replace the ChatGPT component with Llama 2. My understanding is that Llama 2 should be similar to or slightly better than GPT-3.5, so in my fantasy I can have a dedicated server running a possibly fine-tuned version of Llama 2 that will enable me to use a lot more tokens - grade on more dimensions, have more interactivity with the users, and so on. This probably won't happen though due to the cost of running such a server.