4 ms·
I have my own challenge: I think LLMs can do everything that a human can do and typically way better if the context required for the problem can fit in 10,000 t
by simianwords 8mo ago
I have my own challenge: I think LLMs can do everything that a human can do and typically way better if the context required for the problem can fit in 10,000 tokens.
For now this challenge is text only.
Can we think of anything that LLMs can’t do?
- badgersnake 8mo ago* code * write interesting prose * generate realistic images
- simianwords 8mo agoIt can do all of them. I also said text only.
- deleted 8mo ago[deleted]
- infecto 8mo ago> Only really dumb people think that. Or maybe you are an LLM. You deleted it but still come on. Why would you even think to write that?
- seanhunter 8mo agoThis is a “no true scotsman” challenge. People are going to say llms can’t do certain things and you are going to say they can. Not very interesting.
- simianwords 8mo agoLet’s ask in good faith. Can you suggest something that it can’t do? Functional things. I’ll reply in good faith and consider it.
- seanhunter 8mo agoSay I suggest something : Play a valid game of chess at club level (elo approx 1200 say) using algebraic notation. Then you’re either going to say it can or you’re going to say that requires more than 10000 tokens. This isn’t an interesting conversation and I don’t think you are presenting this challenge in good faith for the reason I gave above.
- simianwords 8mo agohttps://chessbenchllm.onrender.com https://chessbenchllm.onrender.com There are several models with greater than 1200 elo Also https://dubesor.de/chess/chess-leaderboard https://dubesor.de/chess/chess-leaderboard
- psvv 8mo agoI'll admit that's better than I expected, but these ratings also imply there are plenty of humans who will beat LLMs at chess.
- stanford_labrat 8mo agoevery few months i like to ask chatgpt to do the "thinking" part of my job (scientist) and see how the responses stack up. at the beginning 2022 it was useless because the output was garbage (hallucinations and fake data). nowadays its still useless, but for different reasons. it just regurgitates things already known and published and is unable to come up with novel hypotheses and mechanisms and how to test them. which makes sense, for how i understand LLMs operate.
- simianwords 8mo agoIt is used in pure math research already
- stanford_labrat 8mo agosadly it looks like seanhunter was correct, shame.
- logicchains 8mo agoThey can't beat even a mediocre chess player at chess.
- am17an 8mo agoSure. “Tell me a joke”
- qsera 8mo ago> For now this challenge is text only. That is like saying, "My program is better than any human, but binary inputs only!".. Even restricted to text, the LLMs are not better than a human who is expert in a domain. Try talking to it regarding any topic. Even in topics that I am not an expert it, the responses from LLMs quickly becomes bland and uninteresting and devoid of additional information.. This is true even in tech things that after a certain point, I stop talking to it and search for a blog/post/so answer written by a human, which if found, would immediately break the plateau of progress that I was facing with the LLM.
- rhubarbtree 8mo agoLLMs without tooling can not do most tasks. It’s the tools and skills that enable them to do things. So you may as well ask “is there anything a python script can’t do”. It’s just not a meaningful question.