4 ms·
Reinforcement learning from self-play/AlphaWhatever? Nah must just be datasets. :)
by ctoth 1y ago
Reinforcement learning from self-play/AlphaWhatever? Nah must just be datasets. :)
- grumpopotamus 1y agohttps://en.wikipedia.org/wiki/TD-Gammon https://en.wikipedia.org/wiki/TD-Gammon
- Y_Y 1y agoYou raise a really interesting point. I'm sure it's just missed my notice, but I'm not familiar with any projects from antediluvian AI that have been resurrected to run on modern hardware and see where they'd really asymptote if they'd had the compute they deserved.
- FeepingCreature 1y agoTo be fair, usually those projects would need considerable work to be ported to modern multicore machines, let alone GPUs.
- genewitch 1y agocan you name a couple so i can see how much work is involved? markov chains compile fast and respond fast, sure, and neural nets train pretty quick too, so i'm wondering where the cutoff is; expert systems?
- FeepingCreature 1y agoOh I have no idea, sorry if I gave the impression I have special knowledge here. I'm just deriving this from consumer multicore systems being historically rare and GPGPUs being nonexistant.
- rsfern 1y agoThis paper “were RNNs all we needed?” explores this hypothesis a bit, finding that some pre-transformer sequence models can match transformers when trained at appropriate scale. Though they did have to make some modifications to unlock more parallelism https://arxiv.org/abs/2410.01201 https://arxiv.org/abs/2410.01201
- zahlman 1y agoFor that matter, https://gist.github.com/deebs67/8fbcf8b127a63e70d4a3f8590c97701d https://gist.github.com/deebs67/8fbcf8b127a63e70d4a3f8590c97... .
- NitpickLawyer 1y agoAnd architecture stuff like actually useful long context. Whatever they did with gemini 2.5 is miles ahead in long context useful results compared to the previous models. I'd be very surprised if gemini 2.5 is "just" gemini 1 w/ better data.
- shwouchk 1y agoi dont know what all the hype is with gemini 2.5, at least the currently running instance. from my experience at least in conversation mode, it cannot remember my instructions to avoid apologies and similar platitudes from either the “persona”, personal instructions, or from ine message to the next.
- nyrikki 1y agoBig difference between a perfect information, completely specified zero sum game and the real world. As a simple analogy, read out the following sentence multiple times, stressing a different word each time. "I never said she stole my money" Note how the meaning changes and is often unique? That is a lens I to the frame problem and it's inverse, the specification problem. The above problem quickly becomes tower-complete, and recent studies suggest that RL is reinforcing or increasing the weight of existing patterns. As the open domain frame problem and similar challenges are equivalent to HALT, finding new ways to extract useful information will be important for generalization IMHO. Synthetic data is useful, but not a complete solution, especially for tower problems.
- genewitch 1y agoThe one we use is "I always pay my taxes" and as far as synthetic vs real data, there's a lot of gaps in LLM knowledge; and vision models suffer from "limited tags", which used to have workarounds with textual embeddings and the like, but those went by the wayside as LoRA, controlnet, etc. appeared. There's people who are fairly well known that LLMs have no idea about. There's things in books i own that the AI confidently tells me are either wrong or don't exist. That one page about compressing 1 gig wikipedia as small as possible implicitly and explicitly states that AI is "basically compression" - and if the data isn't there, it's not in the compressed set (weights) either. And i'll reply to another comment here, about "24/7 rolling/ for looped" AI - i thought of doing this when i first found out about LLMs, but context windows are the enemy, here. I have a couple of ideas about how to have a continuous AI, but i don't have the capital to test it out.
- energy123 1y agoSelf-play gives you a large explosion of data.