4 ms·
I think these models would get interesting at extreme scale. Generate a novel in 40 iterations on a rack of GPUs. At some point in the future, you will be able
by gdiamos 2y ago
I think these models would get interesting at extreme scale. Generate a novel in 40 iterations on a rack of GPUs.
At some point in the future, you will be able to autogen a 10M line codebase in a few seconds on a giant GPU cluster.
- gdiamos 2y agoDiffusion LLMs also follow scaling laws - https://proceedings.neurips.cc/paper_files/paper/2023/file/35b5c175e139bff5f22a5361270fce87-Paper-Conference.pdf https://proceedings.neurips.cc/paper_files/paper/2023/file/3...
- esperent 2y agoIs it possible that combining multiple AIs will be able to somewhat bypass scaling laws, in a similar way that multicore CPUs can somewhat bypass the limitations of a single CPU core?
- gdiamos 2y agoI’m sure there are ways of bypassing scaling laws, but I think we need more research to discover and validate them
- impossiblefork 2y agoThose aren't the modern type with discrete masking based diffusion though. Of course, these too will have scaling laws.
- nthingtohide 2y agoI read a wikipedia article of a person who was very intelligent but also suffered from a mental illness. He told people around him that his next novel will be of exactly N number of words and it will end with the sentence P. I don't remember article. I read it a decade ago. It's like he was doing diffusion in his mind, subconsciously perhaps
- eru 2y agoSeems pretty easy to achieve if you have text editing software that tells you the number of words written so far?