4 ms·
Truthful Language Models as a warm-up for aligned AGI
- PaulHoule 5y agoIt sounds to me like "the cure to alcoholism is having another drink". The truth is out there, it's not in the model. The model can have a database of facts and check that the output is consistent with the facts but if you want to know the truth for many cases you need to check the outside world, take some measurements, do an experiment, etc. I like the idea of "mixture of experts" and having a proofreader that catches the system when it makes critical mistakes. This is how IBM Watson got superhuman performance at jeopardy and both of those are widely known by people in the research triangle park are who work on projects for the government but seem to be unheard of on the west coast. Today when you have a neural network everything looks like visual recognition but I'd like to remind you that replacing bools with floats and taking derivatives doesn't repeal fundamental results of math, logic and computer science such as those of Turing and Gödel. This points to the structural inadequacy of today's neural systems: it was clear in the 1970s that many 'intelligent' tasks require SMT solving. It's not a complete paradigm for intelligence but people do it consciously when they really want to make sure their stories make sense. Truth itself is a problematic concept. Many statements can't be assigned a true or false value for one reason or another. Most "true" or "false" values are probabilistic and worse yet, can only be correctly modeled as part of a joint probability distribution (e.g. no probability for a given statement, rather a probability for assignments of all of the statements.) Go read Godel, Escher, Bach and then some harder stuff, lose the long-winded explanations of why people should care, rewrite it and you might get a proposal worth submitting after a few passes.