5 ms·

I'm using Pro. It's definitely a "hand it to the team and have them schedule a meeting to get back to me" speed tool. But, it "feels" better to me than o3, and

I'm using Pro. It's definitely a "hand it to the team and have them schedule a meeting to get back to me" speed tool. But, it "feels" better to me than o3, and significantly better than gemini/claude for that use case. I do trust it more on confabulations; my current trust hierarchy would be o3-pro -> o3 -> gemini -> claude opus -> (a bunch of stuff) -> 4o. That said, I'd like this quality with a relatively quick tool using model; I'm not sure what else I'd want to call it "AGI" at that point.

agambrahma 1y ago
Hmm, Gemini + O3 > Claude-Opus for ... what kinds of things?
bananapub 1y ago
what do you trust it to do? the only example uses I see written about on HN appear to basically be Substack users asking o3 marketing questions and then writing substack posts about it, and a smattering of vague posts about debugging.
- vessenes 1y ago
  Long form research reporting. Example: Pull together a list of the top 20 startups funded in Germany this year, valuation, founder and business model. Estimate which is most likely to want to take on private equity investment from a lower mid market US PE fund, as well as which would be most suitable taking into consideration their business model, founders and market; write an approach letter in english and in german aimed at getting a meeting. make sure that it's culturally appropriate for german startup founders. I have no idea what the output of this query would be by the way, but it's one I would trust to get right on * the list of startups * the letter and its cultural sensitivity * broad strokes of what the startup is doing Stuff I'd "trust but verify" would be * Names of the founders * Size of company and target market Stuff I'd double check / keep my own counsel on * Suitability and why (note that o3 pro is def. better at this than o3 which is already not bad; it has some genuinely novel and good ideas, but often misses things.)
  - leptons 1y ago
    This is all stuff I would expect an LLM to "hallucinate" about. Every bit of it.
    - steveklabnik 1y ago
      These tools can search the web to find this kind of data, and show you what they searched. Double checking is essential because hallucinations are still possible, but it's not like in the past where it would just try to make up the data from its training set. That said, it also may find bad data and give you a summary of that, which isn't a direct hallucination, but can still be inaccurate. This is why checking the sources is helpful too.
      - majormajor 1y ago
        I wouldn't expect it to hallucinate but how do you evaluate it's ability to distinguish spam from good info? I.e. the "the first four pages of google results is all crap nowdays" problem.
        steveklabnik 1y ago
        By looking at the pages it looked at and deciding for yourself, just like you would with a web search you invoked yourself. I’ve generally found it to use trustworthy stuff like Stack Overflow, Wikipedia, and university websites. But I also haven’t used it in this way that much or for very serious things. I’d imagine more obscure questions are more likely to end up involving less trustworthy sites.
        leptons 1y ago
        >By looking at the pages it looked at and deciding for yourself, just like you would with a web search you invoked yourself. Or you could just cut out the middleman(bot) and just do the search yourself, since you're going to have to anyway to verify what the "AI" wrote. It's just all so stupid that society is rushing towards this iffy-at-best technology when we still need to do the same work anyway to verify it isn't bullshitting us. Ugh, I hate this timeline.
- lovich 1y ago
  I’ve been using it in my job search by handing it stuff like the hn whose hiring threads, giving it a list of criteria i care about, and have it scour those posts for matching jobs, and then chase down all the companies posting and see if they have anything on their corporate site matching my descriptions. Then I have it take those matches and try and chase down the hiring manager based on public info. I did it at first just to see if it was possible, but I am getting direct emails that have been accurate a handful of times and I never would have gotten that on my own
  - bananapub 1y ago
    This is a good data point - I guess another dimension is incompleteness-tolerance. An LLM is absolutely going to miss some but for your case that doesn’t matter very much. Thank you!
- lukeschlather 1y ago
  I don't think it's necessarily a question of trust, it's a question of cost/benefit, and I can apply this just as much to myself. I have been using a lot more SQL queries lately when I use ChatGPT, because I trust it pretty well to write gnarly queries with subqueries and CASE statements. Things that I wouldn't write myself because it's not worth the time to make the query correct, but ChatGPT can do it in seconds. I had an example where o1 really wowed me - something I don't want to post on the internet because I want to use it to test models. In that case I was thinking through a problem where I had made an incorrect mathematical assumption. I explained my reasoning to o1 and it was able to point out the flaw in my reasoning, along with some examples mathematical expressions that disproved my thinking. The funny thing in this case it basically functioned as a rubber duck. When it started producing a response I had deduced essentially what it told me - but it was pretty nice to see the detailed reasoning with examples that might've taken me a few more minutes to work out. And I never would've produced a little report explaining in detail why I was wrong, I would've just adjusted my thinking. Having the report was helpful.
Y_Y 1y ago
Are you setting the "reasoning effort"? I find going from the default (medium) to high makes a big difference on coding tasks for openai reasoning models.
- pas 1y ago
  what/how does that work internally?
  - Y_Y 1y ago
    https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/reasoning?tabs=python-secure%2Cpy#reasoning-effort https://learn.microsoft.com/en-us/azure/ai-services/openai/h... It's just the number of tokens it's willing to expend in the little internal dialogue before it has to spit out an answer.
qwertox 1y ago
What are you using it for? It's not like that wouldn't matter. With coding using anything is always a hit and miss, so I prefer to have faster models where I can throw away the chat if it turns into an idiot. Would I wait 15 minutes for a transcription from Python to Rust if I don't know what the result will be? No. Would I wait 15 minutes if I'd be a mathematician working on some kind of proof? Probably yes.
- AaronAPU 1y ago
  I feed most of my questions/code to 4o, Gemini, o3-pro (in that order). By the time I’ve read through 4o, Gemini is ready. Etc. It’s the progressive jpg download of 2025. You can short circuit after the first model which gives a good enough response.
  - plufz 1y ago
    How do you reason about the energy consumption/climate impact of feeding the same question to three models? Im not saying there is a clear answer here, would just be interesting to hear your thinking.
    - dfsegoat 1y ago
      It's a tough question and I do things the same way. I feel like we are in awkward phase of: "We know this has severe environmental impact - but we need to know if these tools are actually going to be useful and worth adopting..." - so it seems like just keeping the environmental question at the forefront will be important as things progress.
      - plufz 1y ago
        I agree, we are totally in an exploratory phase. I think its always a little scary when so much money and big companies are involved though.
    - true_religion 1y ago
      How much energy does an AI model use during inferencing versus a human being? This is a rhetorical question. Sure we aren’t capturing every last externality, but optimization of large systems should be pushed toward the creators and operators of those systems. Customers shouldn’t have to validate environmental impact every time they spend 0.05 dollars to use a machine.
achierius 1y ago
Ideally it should be able to do things outside of the realm of programming with strong reliability (at least as strong as human experts), as well as be able to pick up new skills and learn new facts dynamically.
IamLoading 1y ago
The time o3 pro takes is so annoying. I still need some time to get used to that.
JamesBarney 1y ago
I haven't tested o3-pro yet enough to have a good hierarchy of confabulation. I use AI a lot to double check my code via a code review what I've found is Gemini - really good at contextual reasoning. Doesn't confabulate bugs that don't exist. Is really good at finding issues related to large context. (this method calls this method, and it does it with a value that could be this) Sonnet/Opus - Seems to be the more creative. More likely to confabulate bugs that don't exist, but also most likely to catch a bug o3 and gemini missed. o3 - Somewhere in the middle
lelanthran 1y ago
> That said, I'd like this quality with a relatively quick tool using model; I'm not sure what else I'd want to call it "AGI" at that point. Am I the only one who is looking around at the AI industry and seeing only developers and artists being replaced with AI? It's hardly AGI when it can't replace a salesperson, or an accountant, or a lawyer, or a teacher, or ... All the headlines I am seeing are software development related. In this post, you yourself are using s/ware development as a measure for how good/bad AI is.

agambrahma 1y ago

Hmm, Gemini + O3 > Claude-Opus for ... what kinds of things?

bananapub 1y ago

what do you trust it to do? the only example uses I see written about on HN appear to basically be Substack users asking o3 marketing questions and then writing substack posts about it, and a smattering of vague posts about debugging.

vessenes 1y ago
Long form research reporting. Example: Pull together a list of the top 20 startups funded in Germany this year, valuation, founder and business model. Estimate which is most likely to want to take on private equity investment from a lower mid market US PE fund, as well as which would be most suitable taking into consideration their business model, founders and market; write an approach letter in english and in german aimed at getting a meeting. make sure that it's culturally appropriate for german startup founders. I have no idea what the output of this query would be by the way, but it's one I would trust to get right on * the list of startups * the letter and its cultural sensitivity * broad strokes of what the startup is doing Stuff I'd "trust but verify" would be * Names of the founders * Size of company and target market Stuff I'd double check / keep my own counsel on * Suitability and why (note that o3 pro is def. better at this than o3 which is already not bad; it has some genuinely novel and good ideas, but often misses things.)
- leptons 1y ago
  This is all stuff I would expect an LLM to "hallucinate" about. Every bit of it.
  - steveklabnik 1y ago
    These tools can search the web to find this kind of data, and show you what they searched. Double checking is essential because hallucinations are still possible, but it's not like in the past where it would just try to make up the data from its training set. That said, it also may find bad data and give you a summary of that, which isn't a direct hallucination, but can still be inaccurate. This is why checking the sources is helpful too.
    - majormajor 1y ago
      I wouldn't expect it to hallucinate but how do you evaluate it's ability to distinguish spam from good info? I.e. the "the first four pages of google results is all crap nowdays" problem.
      - steveklabnik 1y ago
        By looking at the pages it looked at and deciding for yourself, just like you would with a web search you invoked yourself. I’ve generally found it to use trustworthy stuff like Stack Overflow, Wikipedia, and university websites. But I also haven’t used it in this way that much or for very serious things. I’d imagine more obscure questions are more likely to end up involving less trustworthy sites.
        leptons 1y ago
        >By looking at the pages it looked at and deciding for yourself, just like you would with a web search you invoked yourself. Or you could just cut out the middleman(bot) and just do the search yourself, since you're going to have to anyway to verify what the "AI" wrote. It's just all so stupid that society is rushing towards this iffy-at-best technology when we still need to do the same work anyway to verify it isn't bullshitting us. Ugh, I hate this timeline.
lovich 1y ago
I’ve been using it in my job search by handing it stuff like the hn whose hiring threads, giving it a list of criteria i care about, and have it scour those posts for matching jobs, and then chase down all the companies posting and see if they have anything on their corporate site matching my descriptions. Then I have it take those matches and try and chase down the hiring manager based on public info. I did it at first just to see if it was possible, but I am getting direct emails that have been accurate a handful of times and I never would have gotten that on my own
- bananapub 1y ago
  This is a good data point - I guess another dimension is incompleteness-tolerance. An LLM is absolutely going to miss some but for your case that doesn’t matter very much. Thank you!
lukeschlather 1y ago
I don't think it's necessarily a question of trust, it's a question of cost/benefit, and I can apply this just as much to myself. I have been using a lot more SQL queries lately when I use ChatGPT, because I trust it pretty well to write gnarly queries with subqueries and CASE statements. Things that I wouldn't write myself because it's not worth the time to make the query correct, but ChatGPT can do it in seconds. I had an example where o1 really wowed me - something I don't want to post on the internet because I want to use it to test models. In that case I was thinking through a problem where I had made an incorrect mathematical assumption. I explained my reasoning to o1 and it was able to point out the flaw in my reasoning, along with some examples mathematical expressions that disproved my thinking. The funny thing in this case it basically functioned as a rubber duck. When it started producing a response I had deduced essentially what it told me - but it was pretty nice to see the detailed reasoning with examples that might've taken me a few more minutes to work out. And I never would've produced a little report explaining in detail why I was wrong, I would've just adjusted my thinking. Having the report was helpful.

Y_Y 1y ago

Are you setting the "reasoning effort"? I find going from the default (medium) to high makes a big difference on coding tasks for openai reasoning models.

pas 1y ago
what/how does that work internally?
- Y_Y 1y ago
  https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/reasoning?tabs=python-secure%2Cpy#reasoning-effort https://learn.microsoft.com/en-us/azure/ai-services/openai/h... It's just the number of tokens it's willing to expend in the little internal dialogue before it has to spit out an answer.

qwertox 1y ago

What are you using it for? It's not like that wouldn't matter. With coding using anything is always a hit and miss, so I prefer to have faster models where I can throw away the chat if it turns into an idiot. Would I wait 15 minutes for a transcription from Python to Rust if I don't know what the result will be? No. Would I wait 15 minutes if I'd be a mathematician working on some kind of proof? Probably yes.

AaronAPU 1y ago
I feed most of my questions/code to 4o, Gemini, o3-pro (in that order). By the time I’ve read through 4o, Gemini is ready. Etc. It’s the progressive jpg download of 2025. You can short circuit after the first model which gives a good enough response.
- plufz 1y ago
  How do you reason about the energy consumption/climate impact of feeding the same question to three models? Im not saying there is a clear answer here, would just be interesting to hear your thinking.
  - dfsegoat 1y ago
    It's a tough question and I do things the same way. I feel like we are in awkward phase of: "We know this has severe environmental impact - but we need to know if these tools are actually going to be useful and worth adopting..." - so it seems like just keeping the environmental question at the forefront will be important as things progress.
    - plufz 1y ago
      I agree, we are totally in an exploratory phase. I think its always a little scary when so much money and big companies are involved though.
  - true_religion 1y ago
    How much energy does an AI model use during inferencing versus a human being? This is a rhetorical question. Sure we aren’t capturing every last externality, but optimization of large systems should be pushed toward the creators and operators of those systems. Customers shouldn’t have to validate environmental impact every time they spend 0.05 dollars to use a machine.

achierius 1y ago

Ideally it should be able to do things outside of the realm of programming with strong reliability (at least as strong as human experts), as well as be able to pick up new skills and learn new facts dynamically.

IamLoading 1y ago

The time o3 pro takes is so annoying. I still need some time to get used to that.

JamesBarney 1y ago

I haven't tested o3-pro yet enough to have a good hierarchy of confabulation. I use AI a lot to double check my code via a code review what I've found is Gemini - really good at contextual reasoning. Doesn't confabulate bugs that don't exist. Is really good at finding issues related to large context. (this method calls this method, and it does it with a value that could be this) Sonnet/Opus - Seems to be the more creative. More likely to confabulate bugs that don't exist, but also most likely to catch a bug o3 and gemini missed. o3 - Somewhere in the middle

lelanthran 1y ago

> That said, I'd like this quality with a relatively quick tool using model; I'm not sure what else I'd want to call it "AGI" at that point. Am I the only one who is looking around at the AI industry and seeing only developers and artists being replaced with AI? It's hardly AGI when it can't replace a salesperson, or an accountant, or a lawyer, or a teacher, or ... All the headlines I am seeing are software development related. In this post, you yourself are using s/ware development as a measure for how good/bad AI is.