4 ms·
I just thought of a good test. Anyone have feedback? We completely remove a couple simple, obvious inventions from the training data and then see if the AI can
by bilsbie 1y ago
I just thought of a good test. Anyone have feedback?
We completely remove a couple simple, obvious inventions from the training data and then see if the AI can come up with it. Perhaps a toothbrush for example. Or a comb? But there could be better examples that would also have minimal effect on the final Ai.
Training is expensive so we wouldn’t want to leave anything important out like the wheel.
- throwuxiytayq 1y agoOk, you do it. Here’s the internet: https://internet https://internet Make sure you don’t miss any references while you’re combing through, though.
- bilsbie 1y agoI see your point but off the top of my head: a simple regex on each document for a list of dental related words that then gets earmarked for a small LLM to determine if it includes a toothbrush concept.
- throwuxiytayq 1y agoI forgot to mention you’ll have to do this for every language and every possible phrasing. Good luck.
- lightbendover 1y ago[dead]
- ben_w 1y agoIlya Sutskever suggested the same basic idea but for testing for consciousness. I have no idea why this is a PDF, but here's a transcript: https://ecorner.stanford.edu/wp-content/uploads/sites/2/2023/04/a-test-for-ai-consciousness-transcript-2.pdf https://ecorner.stanford.edu/wp-content/uploads/sites/2/2023...
- fsh 1y agoLLM companies try to optimize their benchmark results, not to test the capabilities of their systems. This is why all the benchmarks are so utterly useless.
- thorum 1y agoIt’s very, very hard to remove things from the training data and be sure there is zero leakage. Another idea would be to use, for example, a 2024 state of the art model to try to predict discoveries or events from 2025.