There’s no meaning to the tokens, but research has shown that the models themselves capture meaning. Technically they are producing the next word but in order t

There’s no meaning to the tokens, but research has shown that the models themselves capture meaning. Technically they are producing the next word but in order to do that for a dataset of a trillion words they actually have to develop internal models of how the world works. There was a post on HN a couple days ago that talked about the research done to show this.