If you look at the benchmarks used to evaluate these LLMs, you will find that they contain many questions that test semantic understanding. For example, the bes

If you look at the benchmarks used to evaluate these LLMs, you will find that they contain many questions that test semantic understanding. For example, the best LLMs are good at explaining why novel jokes are funny. Given that humor is notoriously difficult for humans to understand I’d say this is as good an example as any of how there’s more going on under the hood here than a fancy next word predictor would imply.