5 ms·

I open ChatGPT, and I ask GPT-4o: “How many Rs are in the word przepraszam?” It answers: “There are three "R"s in the word "przepraszam.”” It’s interesting t

• astrange 2y ago
  "Begins with" and "contains" are different questions and it should be better at "begins with". On the other hand, it can't do "ends with" at all because LLMs can't reason backwards without chain of thought. Which it could do, it could put every country's names into its context and then look at those, tokenization issues aside.
  • thatjoeoverthr 2y ago
    It would need a bit more than CoT; in this case probably it would actually need to write code, find a dataset and run the code. ChatGPT (the consumer product, not the model) can do part of this, but the model alone will tend to fail as it simply doesn’t have most spellings. A bigger meta problem is that the model doesn’t know if it knows something or not, so letting a model provide information from its “gut” is catastrophically unreliable. It can’t even know if it should look something up or not, except with a stereotype of itself, which will also be very sloppy. And it’s on top of Google because important people making decisions don’t understand this and think these are “bugs” that will be “fixed”. I use Kagi now