4 ms·
What about actively obtained data - models seeking data, rather than being fed. Human babies put things in their mouths, they try to stand and fall over. They “
by cadamsdotcom 1y ago
What about actively obtained data - models seeking data, rather than being fed. Human babies put things in their mouths, they try to stand and fall over. They “do stuff” to learn what works. Right now we’re just telling models what works.
What about simulation: models can make 3D objects so why not give them a physics simulator? We have amazing high fidelity (and low cost!) game engines that would be a great building block.
What about rumination: behind every Cursor rule for example, is a whole story of why a user added it. Why not take the rule, ask a reasoning model to hypothesize about why that rule was created, and add that rumination (along with the rule) to the training data. Providing opportunities to reflect on the choices made by their users might deepen any insights, squeezing more juice out of the data.
- kevmo314 1y agoThat would be reinforcement learning. The juice is quite hard to squeeze.
- cadamsdotcom 1y agoAgreed for most cases. Each Cursor rule is a byproduct of tons of work and probably contains lots that can be unpacked. Any research on that?
- kevmo314 1y agoYeah, at a very high level it's similar to an actor-critic reinforcement learning algorithm. The rule text is a value function and one could build a critic model that takes as input the rule text and the main model's (the actor's) output to produce a reward. This is easier said than done though because this value function is so noisy it's often hard to learn from it. And also whether or not a response (the model output) matches the value function (the Cursor rules) is not even that easy to grade. It's been easier to train the chain-of-thought style reasoning since one can directly score it via the length of thinking. This new paper covers some of the difficulties of language-based critic models: https://openreview.net/pdf?id=0tXmtd0vZG https://openreview.net/pdf?id=0tXmtd0vZG Generally speaking, the algorithm and approach is not new. Being able to do it in a reasonable amount of compute is the new part.
- cadamsdotcom 1y agoSuggestion was even simpler - feed a reasoning model a prompt like “tell me a few reasons a user might’ve created this Cursor rule: {RULE_TEXT}” Do that for a bunch of rules scraped from a bunch of repos - and you’ve got yourself a dataset for training a new model with - or maybe for fine tuning.
- kevmo314 1y agoYeah, go for it.
- Centigonal 1y agoSimulation and embodied AI (putting the AI in a robotic arm or a car so it can try stuff and gather information about the results) are very actively being explored.
- cadamsdotcom 1y agoWhat about at inference time? ie. in response to a query. We let models write code and run it. Which gives them a high chance of getting arithmetic right. Solving the “crossing the river” problem by letting the model create and run a simulation would give a pretty high chance of getting it right.
- Centigonal 1y agoThe newest Claude update comes with a python sandbox built right into the API for exactly this reason. https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/code-execution-tool https://docs.anthropic.com/en/docs/agents-and-tools/tool-use...