27 ms·

Just did a test last week and OpenAIs research was way better. Found 10x more sources and did an overall pretty great job The task was to lookup information ab

by nico 2y ago

Just did a test last week and OpenAIs research was way better. Found 10x more sources and did an overall pretty great job The task was to lookup information about a late distant family member who had been a prominent employee in a certain foreign government about 100 years ago Gemini barely scratched the surface and pretty much gave up ChatGPT on the other hand, kept building up on its research, connecting the dots and leveraging each bit of acquired information to try to find more

consumer451 2y ago
Would love to see this repeated with this latest version from Google. Man, what's really missing from all of this is a 3rd party AI Consumer Reports type site for all of these LLM tools. Whoever does this thing that does not scale will have a highly referenced site on their hands.
- jeffbee 2y ago
  Throughout the entire 20th century the main determinant of a Consumer Reports rating for a car was whether you could put a wheelchair in the trunk. Hopefully the AI agent industry does not sprout a similarly worthless metric.
  - consumer451 2y ago
    I almost didn't use that as the comparison for their lack of rigor, but it gets the idea across.
- shigawire 2y ago
  Isn't that what llmarena does?
  - consumer451 2y ago
    It tries to in a way that scales easily, and is also easily gamed. I want a staff of human testers, each with domain expertise. If the goal is to replace humans, should there not be a real human metric? I want a physicist asking their battery of physics questions, 4 different kinds of devs asking their battery of dev problems, a couple chefs asking for cooking techniques, etc. Now on to "Deep Research," 6 different kinds of OSINT/secondary analysts who ask new problems each time, and compare it to their days of human work. We really need this as a species, otherwise the brain dead C-Suites of the world are going to keep buying the hype, which is often very premature. This could have real consequences, and it apparently already has. It's insane to me that we are investing, what, almost $1T into LLMs, and have not spent the ~$1.5M/year to do what I described above.
    - consumer451 2y ago
      ^ I really should have used the "myopic," instead of "brain dead" to describe the C-Suites of the world. My apologies.
- SequoiaHope 2y ago
  I suppose Consumer Reports could do it!
phonon 2y ago
But this is a brand new version? Why not run it again on Gemini 2.5 Deep Research mode and report if it's better?
infecto 2y ago
I do think they are leaps in front of everyone else from the product perspective and everyday its looking more to be the battleground where money is going to be made.

by nico 2y ago

consumer451 2y ago

Would love to see this repeated with this latest version from Google. Man, what's really missing from all of this is a 3rd party AI Consumer Reports type site for all of these LLM tools. Whoever does this thing that does not scale will have a highly referenced site on their hands.

jeffbee 2y ago
Throughout the entire 20th century the main determinant of a Consumer Reports rating for a car was whether you could put a wheelchair in the trunk. Hopefully the AI agent industry does not sprout a similarly worthless metric.
- consumer451 2y ago
  I almost didn't use that as the comparison for their lack of rigor, but it gets the idea across.
shigawire 2y ago
Isn't that what llmarena does?
- consumer451 2y ago
  It tries to in a way that scales easily, and is also easily gamed. I want a staff of human testers, each with domain expertise. If the goal is to replace humans, should there not be a real human metric? I want a physicist asking their battery of physics questions, 4 different kinds of devs asking their battery of dev problems, a couple chefs asking for cooking techniques, etc. Now on to "Deep Research," 6 different kinds of OSINT/secondary analysts who ask new problems each time, and compare it to their days of human work. We really need this as a species, otherwise the brain dead C-Suites of the world are going to keep buying the hype, which is often very premature. This could have real consequences, and it apparently already has. It's insane to me that we are investing, what, almost $1T into LLMs, and have not spent the ~$1.5M/year to do what I described above.
  - consumer451 2y ago
    ^ I really should have used the "myopic," instead of "brain dead" to describe the C-Suites of the world. My apologies.
SequoiaHope 2y ago
I suppose Consumer Reports could do it!

phonon 2y ago

But this is a brand new version? Why not run it again on Gemini 2.5 Deep Research mode and report if it's better?

infecto 2y ago

I do think they are leaps in front of everyone else from the product perspective and everyday its looking more to be the battleground where money is going to be made.