3 ms·

Attention is All You Need. Was not just an esoteric comment about dot-product attention and transformers. It was a proclamation. With teeth. For decades the st

Attention is All You Need. Was not just an esoteric comment about dot-product attention and transformers. It was a proclamation. With teeth. For decades the statistical language folks had to take it from the symbolic AI camp, which made pretty good progress for its time with expert systems. But symbolic systems also stumbled at seemingly trivial tasks, like parsing the following sentence: Time flies like a banana. It’s pretty clear to a human what this means, but for a computer it’s pretty hard. There’s nothing in the knowledge base about “time flies”, nor what this particular species eats. Once computers became powerful enough, statistical language modeling folks just ignored that entire issue and digested the whole internet. Strings that would start a lexical parser on fire, a GPT eats for breakfast. It’s an approach that seems to work pretty well for some things. But statistical language models aren’t the panacea. They have issues. They hallucinate. They bullshit. They lie with such charming bravado that they can con pretty much anyone not suspecting it. In my personal opinion, we won’t have a complete solution without burying the hatchet and marrying the two camps.

sitkack 4y ago
It is almost like you need multiple competing minds to form a consensus and hold each other in check. I really enjoy watching LLMs bullshit, it is like they see a path to a goal and take it. http://www2.csudh.edu/ccauthen/576f12/frankfurt__harry_-_on_bullshit.pdf http://www2.csudh.edu/ccauthen/576f12/frankfurt__harry_-_on_...
legostormtroopr 4y ago
> Time flies like a banana. > It’s pretty clear to a human what this means. Is it though? The full quote is "Time flies like an arrow, fruit flies like a banana". The humour, being the nonsense that the first phrase sets you up with a metaphor, the secod is literal. "Time flies like a banana" means very little, because while "time flies" and "flies like bananas" make sense, as you said "time flies" aren't a thing and banana's can't fly. As a matter of fact, ChatGPT agrees and said '"Time flies like an arrow" is a common idiomatic expression meaning that time seems to pass quickly. The phrase "time flies like a banana" does not have any meaning or usage in the English language. It may be a play on words or a made-up phrase.'

sitkack 4y ago

It is almost like you need multiple competing minds to form a consensus and hold each other in check. I really enjoy watching LLMs bullshit, it is like they see a path to a goal and take it. http://www2.csudh.edu/ccauthen/576f12/frankfurt__harry_-_on_bullshit.pdf http://www2.csudh.edu/ccauthen/576f12/frankfurt__harry_-_on_...

legostormtroopr 4y ago

> Time flies like a banana. > It’s pretty clear to a human what this means. Is it though? The full quote is "Time flies like an arrow, fruit flies like a banana". The humour, being the nonsense that the first phrase sets you up with a metaphor, the secod is literal. "Time flies like a banana" means very little, because while "time flies" and "flies like bananas" make sense, as you said "time flies" aren't a thing and banana's can't fly. As a matter of fact, ChatGPT agrees and said '"Time flies like an arrow" is a common idiomatic expression meaning that time seems to pass quickly. The phrase "time flies like a banana" does not have any meaning or usage in the English language. It may be a play on words or a made-up phrase.'