4 ms·

> Turns out that if you do UXR, even if computer use is 100% successful in the action execution, and there's no latency, people don't use it. (interesting to me

> Turns out that if you do UXR, even if computer use is 100% successful in the action execution, and there's no latency, people don't use it. (interesting to me is, the core demo was buying airline tickets, and so is OpenAI's. no one would defer to a computer on that, for humanist / design reasons) I would never buy plane tickets that way, we built it because there are tons of things we couldn't automate and this was the only way to do it > - You're not going to be able out-do model companies on building models, they have too much funds. There are plenty of people edging out the model companies all over the place. We aren't powerless to them > Note the reasoning models are worse at tool calling. It's very, very, VERY stark when you have Claude next to o1/4o. OpenAI also owns up to this in the o3-mini paper, though its not under a blaring red line headline or phrased that straightforwardly. Why is that? You're fighting against the current when you're trying to teach the next token predictor to throw a bunch of text out there to <think>, then generate perfectly correct JSON/python/whatever given N tools. The reason why they are bad at tool calling is they aren't trained on it. The current reasoning models require hard-verification reward models, we don't have those for tool calling, whereas that data is easy to get for math/code. Reasoning will improve tool calling, OpenAI just talked about it recently as being the answer to autonomous agents

refulgentis 2y ago
> The reason why they are bad at tool calling is they aren't trained on it. Yes they are.* > OpenAI just talked about it recently as being the answer to autonomous agents You nailed it. It's key to autonomous agents, OpenAI says so out loud, and yet, we're 6 months into reasoning models and performance is regressing, which OpenAI also says out loud but in the fine print of a model card. I know this is a flaming hot take because current thing is reasoning. But it's completely mistaken that reasoning models help with tool-use, both in theory and practice, which puts them in quite a situation. I'm sure they'll figure it out, but I'm also sure on a long enough timeline the LLM is a computer. * I have a bad habit of dismissing without evidence that which is asserted without evidence, and 'not even wrong', in the Pauli sense. It's a cheap way to avoid confrontation. But it makes me look petulant. We can observe, inter alia, every release post September 2024 (i.e. o1-mini and o1-preview) can call tools. (i.e. o1, o3-mini)
- mountainriver 2y ago
  Please cite the source for o1 being trained heavily on tool usage. Reasoning will obviously improve agentic workflows, reasoning is the main problem we see today with autonomous agents. It seems to be likely the _last_ issue we have with them
  - refulgentis 2y ago
    Hmm... - It's going to be difficult to find a source that says heavily on tool use. They don't talk about training at all anymore! :( And saying you trained heavily on tool use would imply your model isn't trained well for other stuff, which they'd want to avoid. - Nothing implies they didn't do the exact same tool training they have been. Both model cards mention tool use. - Both o1 and o3-mini have bog-standard tool calling via API. I'm not sure what gives us the sense that they weren't trained on them - The claim shifted significantly: From: "The reason why they are bad at tool calling is they aren't trained on it" To: "[I don't think you can find a source that says] o1 [was] trained heavily on tool usage."

refulgentis 2y ago

> The reason why they are bad at tool calling is they aren't trained on it. Yes they are.* > OpenAI just talked about it recently as being the answer to autonomous agents You nailed it. It's key to autonomous agents, OpenAI says so out loud, and yet, we're 6 months into reasoning models and performance is regressing, which OpenAI also says out loud but in the fine print of a model card. I know this is a flaming hot take because current thing is reasoning. But it's completely mistaken that reasoning models help with tool-use, both in theory and practice, which puts them in quite a situation. I'm sure they'll figure it out, but I'm also sure on a long enough timeline the LLM is a computer. * I have a bad habit of dismissing without evidence that which is asserted without evidence, and 'not even wrong', in the Pauli sense. It's a cheap way to avoid confrontation. But it makes me look petulant. We can observe, inter alia, every release post September 2024 (i.e. o1-mini and o1-preview) can call tools. (i.e. o1, o3-mini)

mountainriver 2y ago
Please cite the source for o1 being trained heavily on tool usage. Reasoning will obviously improve agentic workflows, reasoning is the main problem we see today with autonomous agents. It seems to be likely the _last_ issue we have with them
- refulgentis 2y ago
  Hmm... - It's going to be difficult to find a source that says heavily on tool use. They don't talk about training at all anymore! :( And saying you trained heavily on tool use would imply your model isn't trained well for other stuff, which they'd want to avoid. - Nothing implies they didn't do the exact same tool training they have been. Both model cards mention tool use. - Both o1 and o3-mini have bog-standard tool calling via API. I'm not sure what gives us the sense that they weren't trained on them - The claim shifted significantly: From: "The reason why they are bad at tool calling is they aren't trained on it" To: "[I don't think you can find a source that says] o1 [was] trained heavily on tool usage."