9 ms·

The Deep Research problem

302 points by cratermoon 2y ago

Lws803 2y ago
I always wondered, if deep research has an X% chance of producing errors in it's report and you have to double check everything + visit every source or potentially correct it yourself. Then does it really save time in helping you get research done (outside of coding and marketing)? .
- ImaCake 2y ago
  It might depend on how much you struggle with writers block. An LLM essay with sources is probably a better starting point than a blank page. But it will vary between people.
baxtr 2y ago
I urge anyone to do the following: take a subject you know really really well and then feed it into one of the deep research tools and check the results. You might be amazed but most probably very shocked.
- ilrwbwrkhv 2y ago
  Yup none of these tools are actually any close to AGI or "research". They are still a much better search engine and of course spam generator.
- genewitch 2y ago
  Murrai Gell-Mann amnesia
- caseyy 2y ago
  In my experience, Perplexity and OpenAI's deep research tools are so misleading that they are almost worthless in any area worth researching. This becomes evident if one searches for something they know or tries to verify the facts the models produce. In my area of expertise, video game software engineering, about 80% of the insights are factually wrong cocktail-party-level thoughts. The "deep research" features were much more effective at getting me to pay for both subscriptions than in any valuable data collection. The former, I suspect, was the goal anyway. It is very concerning that people will use these tools. They will be harmed as a result.
  - brokencode 2y ago
    > “They will be harmed as a result.” Compared to what exactly? The ad-fueled, SEO-optimized nightmare that is modern web search? Or perhaps the rampant propaganda and blatant falsehoods on social media? Whoever is blindly trusting what ChatGPT is spitting out is also falling for whatever garbage they’re finding online. ChatGPT is not very smart, but at least it isn’t intentionally deceptive. I think it’s an incredible improvement for the low information user over any current alternatives.
    - simianparrot 2y ago
      It’s deceptive by design because there is no reasoning, and humans created it and know this.
      - brokencode 2y ago
        Clearly it is able to solve various logical problems, and therefore an at least imitate logical thought. Is that not reasoning? And there are plenty of logical problems that many humans can’t solve. Does that mean they’re not capable of reasoning? At what point would you say something has reasoning? I’d argue that it’s more about how good something is at reasoning, rather than saying it is or isn’t capable of reasoning in absolute terms.
    - deleted 2y ago
      [deleted]
    - caseyy 2y ago
      OpenAI knows the tool it markets as “research” does not pass muster. It hallucinates, mid-quotes sources, and does not follow the formal inference logic used in research. AI slop already produces many plausible-sounding articles used as infotainment and in academia. We already know this slop adds much noise to the signal and that poor signal slows actual research in both cases. But until now, the slop wasn't masquerading specifically as research! It was presented as an assistant, which provides no accuracy guarantees. “Research” by the word’s common meanings does. This is why it will do harm. There is no doubt in my mind. And I believe OpenAI knows it. They have quite smart engineers, certainly clever enough to figure it out.
- jsemrau 2y ago
  In my case very "not useful". Background, I am writing a Substack where I write "deep research" articles on autonomous agent tech and explored several of these tools to understand the risks to my workflow, but none of them can replace my as of now.
- aerhardt 2y ago
  I tried to get it to research the design of models for account potential in B2B sales. It went to the shittiest blogspam sites and returned something utterly unimpressive. Instacancelled the $200 sub. Will try it a few more times this month but my expectations are very low.
tptacek 2y ago
I did a trial run with Deep Research this weekend to do a comparative analysis of the comp packages for Village Managers in suburbs around Chicagoland (it's election season, our VM's comp had become an issue). I have a decent idea of where to look to find comp information for a given municipality. But there are a lot of Chicagoland suburbs and tracking documents down for all of them would have been a chore. Deep Research was valuable. But it only did about 60% of the work (which, of course, it presented as if it was 100%). It found interesting sources I was unaware of, and assembled lots of easy-to-get public data that would have been annoying for me to collect that made spot-checking easier (for instance, basic stuff like the name of every suburban Village Manager). But I still had to spot check everything myself. The premise of this post seems to be that material errors in Deep Research results negate the value of the product. I can't speak to how OpenAI is selling this; if the claim is "subscribe to Deep Research and it will generate reliable research reports for you", well, obviously, no. But as with most AI things, if you get paste the hype, it's plain to see the value it's actually generating.
- WhitneyLand 2y ago
  >>The premise of this post seems to be that material errors in Deep Research results negate the value of the product No it’s not. It’s that it’s oversold from a marketing perspective and comes with some big caveats. But it does talk about big time savings for the right contexts. Emphasis from the article: “these things are useful”
- aprilthird2021 2y ago
  I'm just realizing this might finally be something that helps me get past analysis paralysis I have before committing to so many decisions online. I always feel like without doing my research, I'll get scammed. Maybe this will help give me a bit more confidence
  - xmprt 2y ago
    On the flipside, you might end up getting scammed even worse because of incorrect analysis. For example if ChatGPT hallucinates some data/features through faulty research then you might be surprised when you actually make the decision.
iandanforth 2y ago
I'll share my recipe for using these products on the off chance it helps someone. 1. Only do searches that result in easily verifiable results from non-AI sources. 2. Always perform the search in multiple products (Gemini 1.5 Deep Research, Gemini 2.0 Pro, ChatGPT o3-mini-high, Claude 3.7 w/ extended thinking, Perplexity) With these two rules I have found the current round of LLMs useful for "researchy" queries. Collecting the results across tools and then throwing out the 65-75% slop results in genuinely useful information that would have taken me much longer to find. Now the above could be seen as a harsh critique of these tools, as in the kiddie pool is great as long as you're wearing full hazmat gear, but I still derive regular and increasing value from them.
- munchler 2y ago
  This makes sense. How many of those products do you have to pay for?
  - kridsdale3 2y ago
    I'm not OP but I do similar stuff. I pay for Claude's basic tier, OpenAI's $200 tier, and Gemini ultra-super-advanced I get for free because I work there. I combine all the 'slop' from the three of them in to Gemini (1 or 2 M context window) and have it distill the valuable stuff in there to a good final-enough product. Doing so has got me a lot of kudos and applause from those I work with.
    - munchler 2y ago
      Wow, that's eye-opening. So, just to be clear, you're paying for Claude and OpenAI out of your own pocket, and using the results at your Google job? We live in interesting times, for sure. :)
      - deleted 2y ago
        [deleted]
      - kridsdale3 2y ago
submeta 2y ago
Deep Research is in its „ChatGPT 2.0“ phase. It will improve, dramatically. And to the naysayers: When OpenAI released its first models, many doubted that it will be good at coding. Now after two years look at Cursor, aider, and all the llms powering them, what you can do with a few prompts and iterations. Deep research will dramatically improve as it’s a process that can be replicated and automated.
- amelius 2y ago
  This is like saying: y=e^-x+1 will soon be 0, because look at how fast it went through y=2!
  - kridsdale3 2y ago
    I appreciate your style of humor.
  - PeterFBell 2y ago
    Thanks for making my day :)
  - whyenot 2y ago
    Tony Tromba (my math advisor at UCSC) used to tell a low key infuriating, sexist and inappropriate story about a physicist, a mathematician, and a naked woman. It ended with the mathematician giving up in despair and a happy physicist yelling "close enough."
    - rpmisms 2y ago
      This sounds like a joke with a lot of truth, even if it is offensive.
    - john_minsk 2y ago
      Can I have a joke?
    - lynx97 2y ago
      [dead]
lsy 2y ago
Research skills involve not just combining multiple pieces of data, but also being able to apply very subtle skills to determine whether a source is trustworthy, to cross-check numbers where their accuracy is important (and to determine when it's "important"), and to engage in some back and forth to determine which data actually applies to the research question being asked. In this sense, "deep research" is a misleading term, since the output is really more akin to a probabilistic "search" over the training data where the result may or may not be accurate and requires you to spot-check every fact. It is probably useful for surfacing new sources or making syntactic conjectures about how two pieces of data may fit together, but checking all of those sources for existence, let alone validity, still needs to be done by a person, and the output, as it stands in its polished form today, doesn't compel users to take sufficient responsibility for its factuality.
rollinDyno 2y ago
Everyone who has been working on RAG is aware of how important source control is. Simply directing your agent to fetch keyword matching documents will lead to inaccurate claims. The reality is that for now it is not possible to leave the human out of research, so I think the best LLM can only help curate sources and synthesize them, but cannot reliably write sound conclusions. Edit: this is something elicit.com recognized quite early. But even when I was using it, I was wishing I had more control over the space over which the tool was conducting search.
- jslakro 2y ago
  Human-in-the-loop (HITL) a buzzword that has become common these days
theGnuMe 2y ago
One other existential question is Simpson's paradox, which I believe is exploited by politicians to support different policies from the same underlying data. I see this as a problem for government especially if we have liberal or conservative trained LLMs. We expect the computer to give us the correct answer, but when the underlying model is trained one way by RLHF or by systemic/weighted bias in its source documents -- Imagine training a libertarian AI on Cato papers -- you have could have highly confident pseudo-intellectual junk. Economists already deal with this problem daily since their field was heavily politicized. Law as well is another one.
- ImaCake 2y ago
  I've never thought of Simpson's Paradox as a political problem before, thanks for sharing this! Arguably this applies just as well to Bayesian vs Frequentist statisticians or Molecular vs Biochemical Biologists.
  - theGnuMe 2y ago
    What examples do you have for Bayes vs freq; and molecular vs biochem?
jppope 2y ago
These days I'm feeling like GenAi is basically an accuracy rate of 95% maybe 96%. Great at boilerplate, great at stuff you want an intern to do or maybe to outsource... but it really struggles with the valuable stuff. The errors are almost always in the most inconvenient places and they are hard to see... So I agree with Ben Evans on this one, what is one to do? the further you lean on it the worse your skills and specializations get. It is invaluable for some kinds of work greatly speeding you up, but then some of the things you would have caught take you down a rabbit hole that waste so much time. The tradeoffs here aren't great.
- bakari500 2y ago
  Yeah but you have 4 to 6 % error that’s not good even if you have dumb computer
- daxfohl 2y ago
  I think it's not the valuable stuff though. The valuable stuff is all the boilerplate, because, I don't want to do it. The rest, I actually have a stake in not only that it's done, but how it's done. And I'd rather be hands-on doing that and thinking about it as I do it. Having an AI do that isn't that valuable, and in fact robs me of the learning I acquire by doing it myself.
  - deleted 2y ago
    [deleted]
smusamashah 2y ago
Watched recent Viva la dirt league videos on how trailers lie and do false promises. Now I see LLM as that marketing guy. Even if he knows everything, he can't help with lying. You can't trust anything he says no matter how authoritative he sounds, even if he is telling the truth you have know way of knowing. These deep research things are a waste of time if you can't trust the output. Code you can run and verify. How do you verify this.
- deleted 2y ago
  [deleted]
ano-ther 2y ago
Similar to what Derek Lowe found with a pharma example: https://www.science.org/content/blog-post/evaluation-deep-research-performance https://www.science.org/content/blog-post/evaluation-deep-re... > As with all LLM output, all of these things are presented in the same fluid, confident-sounding style: you have to know the material already to realize when your foot has gone through what was earlier solid flooring.
- stogot 2y ago
  Of which, people will surely die (when it is used to publish medical research by those wishing not to fail the publish-vs-perish situation)
  - genewitch 2y ago
    I think the "delve" curve shows were already well into "AI papers" stage of civilization. It has probably been tamped down, now; the last thing j heard using it like a tell was notebookllm. Deep dive. "Yes, that's exactly right"
light_triad 2y ago
"Deep research" is super impressive, but so far is more "search the web and surf pages autonomously to aggregate relevant data" It is in many ways a workaround to Google's SEO poisoning. Doing very deep research requires a lot of context, cross-checking data, resourcefulness in sourcing and taste. Much of that context is industry specific and intuition plays a role in selecting different avenues to pursue and prioritisation. The error rates will go down but for the most difficult research it will be one tool among many rather than a replacement for the stack.
- j_maffe 2y ago
  > It is in many ways a workaround to Google's SEO poisoning. But the article goes into exactly how Deep Research fell exactly for the same SEO traps.
  - light_triad 2y ago
    Right, hence the term "workaround" not "solution"
simonw 2y ago
When ChatGPT came out, one of the things we learned is that human society generally assumes a strong correlation between intelligence and the ability to string together grammatically correct sentences. As a result, many people assumed that even GPT-3.5 was wildly more "intelligent" than it actually was. I think Deep Research (and tools like it) offer an even stronger illustration of that same effect. Anything that can produce a well-formatted multiple page report with headings and citations surely must be of PhD-level intelligence, right? (Clearly not.)
- SubiculumCode 2y ago
  In some ways, it's a good tool to teach yourself to sus out the real clues to reliability, not format and authoritative tone.
  - j_maffe 2y ago
    But that's the thing. The only way to truly find out if it's reliable (>90%) is to check the data yourself.
    - SubiculumCode 2y ago
      This is why metrics and leaderboards like these are so important (but under reported on): https://github.com/vectara/hallucination-leaderboard https://github.com/vectara/hallucination-leaderboard https://www.kaggle.com/facts-leaderboard https://www.kaggle.com/facts-leaderboard Google Gemni models seem to lead...hopefully the metrics aren't being gamed.
- immibis 2y ago
  That's been every LLM since GPT-2.
- isaiahwp 2y ago
  To be fair, OpenAI's the one marketing it as such.
- jcgrillo 2y ago
deleted 2y ago
[deleted]
somerandomness 2y ago
Indeed, the main drawback of the various Deep Research implementation is the quality of sources is determined by SEO, which is often sketchy. Often the user has insight on what the right sources are and they may even be off-line on your computer. We built an alternative to do Deep Research (https://radpod.ai https://radpod.ai) on data you provide, instead of relying on Web results. We found this works a lot better in terms of quality of answers as the user can control the source quality.
- deleted 2y ago
  [deleted]
FeepingCreature 2y ago
> Are you telling me that today’s model gets this table 85% right and the next version will get it 85.5 or 91% correct? That doesn’t help me. If there are mistakes in the table, it doesn’t matter how many there are - I can’t trust it. If, on the other hand, you think that these models will go to being 100% right, that would change everything, but that would also be a binary change in the nature of these systems, not a percentage change, and we don’t know if that’s even possible. Of course, humans also make mistakes. There is a percentage, usually depending on the task but always below 100%, where the work is good enough to use, because that's how human labor works.
- deleted 2y ago
  [deleted]
- semi-extrinsic 2y ago
  If I'm paying a human, even a student working part-time or something, I expect "concrete facts extracted from the literature" to be correct at least 99.99% of the time. There is a huge gap from 85% to 99.99%.
  - anthuswilliams 2y ago
    This does not at all resonate with my experience with human researchers, even highly paid ones. You still have to do a lot of work to verify their claims.
  - FeepingCreature 2y ago
    Sure but there's no step change.
  - nuancebydefault 2y ago
    You can expect that from a human, but if you don't know their reputation, you'd be lucky with the 85 percent. How do you even know if they understood the question correctly, used trusted sources, correct statistical models etc?
- smusamashah 2y ago
jwpapi 2y ago
Yes the confidence tbh is getting a bit out of hand. I see the same thing with coding with our SAAS, once the problems get bigger I find myself more often than not starting to code the old way, even over "fixing ai’s code", because the issues are often too much. Ithink more certainty communication could help. Especially when they talk about docs or 3rd party packages etc. Regularly even Sonnet 3-7 just invents stuff...
skywhopper 2y ago
This article covers something early on that makes the question of “will models get to zero mistakes” pretty easy to answer: No. Even if they do the math right and find the data you ask for and never make any “facts” up, the sources of the data themselves carry a lot of context and connotation about how the data is gathered and what weight you can put on it. If anything, as LLMs become a more common way of ingesting the Internet, the sources of data themselves will start being SEOed to get chosen more often by the LLM purveyors. Add in paid sponsorship, and if anything, trust in the data from these sorts of Deep Research models will only get worse over time.
furyofantares 2y ago
I used deep research with o1-pro to try to fact/sanity check a current events thing a friend was talking about, read the results and followed the links it provided to get further info, and ended up on the verge of going down a rabbit hole that now looks more like a leftist(!) conspiracy theory.
- furyofantares 2y ago
  I didn't want to bring in specifics because I didn't feel like debating the specific thing, so I guess that made this post pretty hard to parse and I should have added more info. I was trying to convey that it had found some sources that, if I came across them naturally, I probably would have immediately recognized as fringe. The sources were threading together a number of true facts into a fringe narrative. The AI was able to get other sources on the true facts, but has no common sense, and I think ended up producing a MORE convincing presentation of the fringe theory than the source of the narrative. It sounded confident and used a number of extra sources to check facts even though the fringe narrative that threaded them all together was only from one site that you'd be somewhat apt to dismiss just by domain name if it was the only source you found.
deleted 2y ago
[deleted]
tippytippytango 2y ago
I think of it as the return to 10 blue links. It searches the web, finds stuff and summarizes it so I can decide which links to click. I ignore the narrative it constructs because it’s probably wrong. Which I forgive because it’s the hazmat suit for the internet I’ve always dreamed of. It gets in the trenches, braves the cookie popups, email signups, inline ads and overly clever web design so I don’t have to. That’s enough to forgive its attempts to create research narratives. But, I hope we can figure out a way to train a heaping spoonful of humility into future models.
- CreepGin 2y ago
  Agreed. Maybe we're moving toward a world where LLMs do all the searching, and "websites" just turn into data-only endpoints made for AI to consume. That'll have other big implications... Interesting times ahead.
  - Seattle3503 2y ago
    What do the websites get out of that exchange?
    - fragmede 2y ago
      therein lies the problem, and is why Google search didn't disrupt itself until ChatGPT came around.
  - ben_w 2y ago
    Interesting times, for sure. > and "websites" just turn into data-only endpoints made for AI to consume. As is already the case with humans, that only serves users to the extent that the websites' veracity is within the intelligence's ability to verify — all the problems we've had with blogspam etc. have been due to the subset of SEO where people abuse* the mechanisms to promote what they happen to be selling at the expense of genuine content. AI generated content is very good at seeming human, at seeming helpful. A "review website" which is all that, but with fake reviews that promote one brand over the others… a chain of such websites that link to each other to boost PageRank scores… which are then cross-linked with a huge number of social media bots… Will lead to a lot of people who think they're making an informed choice, but who were lied to about everything from their cornflakes to their president. * Tautologically, when it's not "abuse", SEO is only helping the search engine find the real content. I've seen places fail to perform any SEO including the legitimate kind.
spoaceman7777 2y ago
I, for one, have it in my prompt that GPT should end every message with a message about how sure it is about the answer, and a rating of "Extremely sure", "moderately sure", etc. It works surprisingly well. It also provides its reasoning about the quality of the sources, etc. (This is using GPT-4o of course, as it's the only mature GPT with web access) I highly recommend adding this to your default prompt.
- afro88 2y ago
  > It works surprisingly well What do you mean by this exactly? That it makes you feel better about what its said, or that its assessment of its answer is actually accurate?
  - spoaceman7777 2y ago
    That's its assessment gives me a good picture of which ways to push with my next question, and which things to ask it to use its web search tool to find more information on. It's a conversation with AI, it's good to know its thought process on how certain it is of its conclusions, as it isn't infallible not is any human.
zeckalpha 2y ago
Two factors to consider: human performance and cost. Plenty of humans regularly make similar mistakes to the one in the Deep Research marketing, with more overhead than an LLM.
visarga 2y ago
The thig is, if you look at all the "Deep Research" benchmark scores, they never claim to be perfect. The problem was plain to see.
franze 2y ago
I am currently in India in a big city doing yoga for the first time as a westerner. I dont Google anything. Google maps, yeah. Google, no. Everything I want to know is much better answered by ChatGpt Deep Research. Ask a Question, Drink a Chai, Get a Great, Prioritised, structured Answer without spam or sifting through ad ridden SEO pages. It is a game changer, and at one point the will get rid of the "drink a chai" wait and it will kill the Google we know now.
nuancebydefault 2y ago
The problem with tools like deep research is that they imply good reasoning skills of the underlying technology. Artificial reasoning clearly exists, but is not refined enough to build these kind of technology on top of it. The reasoning is the fundamental of this system and all on top of it gets very unstable.
kgeist 2y ago
Deep Research, as it currently stands, is a jack of all trades but a master of none. Could this problem be mitigated by building domain-specific profiles curated by experts? For example, specifying which sources to prioritize, what to avoid, and so on. You would select a profile, and Deep Research would operate within its specific constraints, supplemented by expert knowledge.
Alifatisk 2y ago
What a beautiful website
localbolu 2y ago
> they don’t really have products either, just text boxes - and APIs for other people to build products. Isn't this a very valuable product in itself? Whatever happened to the phrase "When there is a gold rush, sell shovels"?
jcgrillo 2y ago
This is such embarrassing marketing from an organization (OpenAI) which presents itself to the world as a "research" entity.. They could have at least photoshopped the screenshot to make it look like it emitted correct information.
mirekrusin 2y ago
Not to take away from the main point of the article, which is true but: It seems to be at intern level according to the author - not bad if you ask me, no? Did he try to proceed as with an intern? ie. was it a dialogue? did he try to drop in this article into prompt and see what comes out? For skeptics my best advise is – do your usual work and at the end drop in your whole work with prompt to find issues etc. – it will be net positive result, I promise. And yes they do get better and it shouldn't get dismissed – the most fascinating part is precisely just that – not even their current state, but how fast they keep improving. One part which always bothers me a bit with this type of arguments – why on earth are we assuming that human does it 100% correctly? Aren't humans also making similar mistakes? IMHO there is some similarity with young geniuses – they get tons of stuff right and it's impressive however total, unexpected failures occur which feel weird – in my opinion it's a matter of focused training similar to how you'd teach a genius. It's worth taking step back and recognizing in how many diverse contexts we're using (like now, today, not in 5 years) models like grok3 or claude3.7 – the goalpost seem to have moved to "beyond any human expert on all subjects".
marcuschong 2y ago
"I can say that these systems are amazing, but get things wrong all the time in ways that matter, and so the best uses cases so far are things where the error rate doesn’t matter or where it’s easy to see." That's probably how we should all be using LLMs.

302 points by cratermoon 2y ago

I always wondered, if deep research has an X% chance of producing errors in it's report and you have to double check everything + visit every source or potentially correct it yourself. Then does it really save time in helping you get research done (outside of coding and marketing)? .

ImaCake 2y ago
It might depend on how much you struggle with writers block. An LLM essay with sources is probably a better starting point than a blank page. But it will vary between people.

baxtr 2y ago

I urge anyone to do the following: take a subject you know really really well and then feed it into one of the deep research tools and check the results. You might be amazed but most probably very shocked.

ilrwbwrkhv 2y ago
Yup none of these tools are actually any close to AGI or "research". They are still a much better search engine and of course spam generator.
genewitch 2y ago
Murrai Gell-Mann amnesia
caseyy 2y ago
In my experience, Perplexity and OpenAI's deep research tools are so misleading that they are almost worthless in any area worth researching. This becomes evident if one searches for something they know or tries to verify the facts the models produce. In my area of expertise, video game software engineering, about 80% of the insights are factually wrong cocktail-party-level thoughts. The "deep research" features were much more effective at getting me to pay for both subscriptions than in any valuable data collection. The former, I suspect, was the goal anyway. It is very concerning that people will use these tools. They will be harmed as a result.
- brokencode 2y ago
  > “They will be harmed as a result.” Compared to what exactly? The ad-fueled, SEO-optimized nightmare that is modern web search? Or perhaps the rampant propaganda and blatant falsehoods on social media? Whoever is blindly trusting what ChatGPT is spitting out is also falling for whatever garbage they’re finding online. ChatGPT is not very smart, but at least it isn’t intentionally deceptive. I think it’s an incredible improvement for the low information user over any current alternatives.
  - simianparrot 2y ago
    It’s deceptive by design because there is no reasoning, and humans created it and know this.
    - brokencode 2y ago
      Clearly it is able to solve various logical problems, and therefore an at least imitate logical thought. Is that not reasoning? And there are plenty of logical problems that many humans can’t solve. Does that mean they’re not capable of reasoning? At what point would you say something has reasoning? I’d argue that it’s more about how good something is at reasoning, rather than saying it is or isn’t capable of reasoning in absolute terms.
  - deleted 2y ago
    [deleted]
  - caseyy 2y ago
    OpenAI knows the tool it markets as “research” does not pass muster. It hallucinates, mid-quotes sources, and does not follow the formal inference logic used in research. AI slop already produces many plausible-sounding articles used as infotainment and in academia. We already know this slop adds much noise to the signal and that poor signal slows actual research in both cases. But until now, the slop wasn't masquerading specifically as research! It was presented as an assistant, which provides no accuracy guarantees. “Research” by the word’s common meanings does. This is why it will do harm. There is no doubt in my mind. And I believe OpenAI knows it. They have quite smart engineers, certainly clever enough to figure it out.
jsemrau 2y ago
In my case very "not useful". Background, I am writing a Substack where I write "deep research" articles on autonomous agent tech and explored several of these tools to understand the risks to my workflow, but none of them can replace my as of now.
aerhardt 2y ago
I tried to get it to research the design of models for account potential in B2B sales. It went to the shittiest blogspam sites and returned something utterly unimpressive. Instacancelled the $200 sub. Will try it a few more times this month but my expectations are very low.

tptacek 2y ago

I did a trial run with Deep Research this weekend to do a comparative analysis of the comp packages for Village Managers in suburbs around Chicagoland (it's election season, our VM's comp had become an issue). I have a decent idea of where to look to find comp information for a given municipality. But there are a lot of Chicagoland suburbs and tracking documents down for all of them would have been a chore. Deep Research was valuable. But it only did about 60% of the work (which, of course, it presented as if it was 100%). It found interesting sources I was unaware of, and assembled lots of easy-to-get public data that would have been annoying for me to collect that made spot-checking easier (for instance, basic stuff like the name of every suburban Village Manager). But I still had to spot check everything myself. The premise of this post seems to be that material errors in Deep Research results negate the value of the product. I can't speak to how OpenAI is selling this; if the claim is "subscribe to Deep Research and it will generate reliable research reports for you", well, obviously, no. But as with most AI things, if you get paste the hype, it's plain to see the value it's actually generating.

WhitneyLand 2y ago
>>The premise of this post seems to be that material errors in Deep Research results negate the value of the product No it’s not. It’s that it’s oversold from a marketing perspective and comes with some big caveats. But it does talk about big time savings for the right contexts. Emphasis from the article: “these things are useful”
aprilthird2021 2y ago
I'm just realizing this might finally be something that helps me get past analysis paralysis I have before committing to so many decisions online. I always feel like without doing my research, I'll get scammed. Maybe this will help give me a bit more confidence
- xmprt 2y ago
  On the flipside, you might end up getting scammed even worse because of incorrect analysis. For example if ChatGPT hallucinates some data/features through faulty research then you might be surprised when you actually make the decision.

iandanforth 2y ago

I'll share my recipe for using these products on the off chance it helps someone. 1. Only do searches that result in easily verifiable results from non-AI sources. 2. Always perform the search in multiple products (Gemini 1.5 Deep Research, Gemini 2.0 Pro, ChatGPT o3-mini-high, Claude 3.7 w/ extended thinking, Perplexity) With these two rules I have found the current round of LLMs useful for "researchy" queries. Collecting the results across tools and then throwing out the 65-75% slop results in genuinely useful information that would have taken me much longer to find. Now the above could be seen as a harsh critique of these tools, as in the kiddie pool is great as long as you're wearing full hazmat gear, but I still derive regular and increasing value from them.

munchler 2y ago
This makes sense. How many of those products do you have to pay for?
- kridsdale3 2y ago
  I'm not OP but I do similar stuff. I pay for Claude's basic tier, OpenAI's $200 tier, and Gemini ultra-super-advanced I get for free because I work there. I combine all the 'slop' from the three of them in to Gemini (1 or 2 M context window) and have it distill the valuable stuff in there to a good final-enough product. Doing so has got me a lot of kudos and applause from those I work with.
  - munchler 2y ago
    Wow, that's eye-opening. So, just to be clear, you're paying for Claude and OpenAI out of your own pocket, and using the results at your Google job? We live in interesting times, for sure. :)
    - deleted 2y ago
      [deleted]
    - kridsdale3 2y ago

submeta 2y ago

Deep Research is in its „ChatGPT 2.0“ phase. It will improve, dramatically. And to the naysayers: When OpenAI released its first models, many doubted that it will be good at coding. Now after two years look at Cursor, aider, and all the llms powering them, what you can do with a few prompts and iterations. Deep research will dramatically improve as it’s a process that can be replicated and automated.

amelius 2y ago
This is like saying: y=e^-x+1 will soon be 0, because look at how fast it went through y=2!
- kridsdale3 2y ago
  I appreciate your style of humor.
- PeterFBell 2y ago
  Thanks for making my day :)
- whyenot 2y ago
  Tony Tromba (my math advisor at UCSC) used to tell a low key infuriating, sexist and inappropriate story about a physicist, a mathematician, and a naked woman. It ended with the mathematician giving up in despair and a happy physicist yelling "close enough."
  - rpmisms 2y ago
    This sounds like a joke with a lot of truth, even if it is offensive.
  - john_minsk 2y ago
    Can I have a joke?
  - lynx97 2y ago
    [dead]

lsy 2y ago

Research skills involve not just combining multiple pieces of data, but also being able to apply very subtle skills to determine whether a source is trustworthy, to cross-check numbers where their accuracy is important (and to determine when it's "important"), and to engage in some back and forth to determine which data actually applies to the research question being asked. In this sense, "deep research" is a misleading term, since the output is really more akin to a probabilistic "search" over the training data where the result may or may not be accurate and requires you to spot-check every fact. It is probably useful for surfacing new sources or making syntactic conjectures about how two pieces of data may fit together, but checking all of those sources for existence, let alone validity, still needs to be done by a person, and the output, as it stands in its polished form today, doesn't compel users to take sufficient responsibility for its factuality.

rollinDyno 2y ago

Everyone who has been working on RAG is aware of how important source control is. Simply directing your agent to fetch keyword matching documents will lead to inaccurate claims. The reality is that for now it is not possible to leave the human out of research, so I think the best LLM can only help curate sources and synthesize them, but cannot reliably write sound conclusions. Edit: this is something elicit.com recognized quite early. But even when I was using it, I was wishing I had more control over the space over which the tool was conducting search.

jslakro 2y ago
Human-in-the-loop (HITL) a buzzword that has become common these days

theGnuMe 2y ago

One other existential question is Simpson's paradox, which I believe is exploited by politicians to support different policies from the same underlying data. I see this as a problem for government especially if we have liberal or conservative trained LLMs. We expect the computer to give us the correct answer, but when the underlying model is trained one way by RLHF or by systemic/weighted bias in its source documents -- Imagine training a libertarian AI on Cato papers -- you have could have highly confident pseudo-intellectual junk. Economists already deal with this problem daily since their field was heavily politicized. Law as well is another one.

ImaCake 2y ago
I've never thought of Simpson's Paradox as a political problem before, thanks for sharing this! Arguably this applies just as well to Bayesian vs Frequentist statisticians or Molecular vs Biochemical Biologists.
- theGnuMe 2y ago
  What examples do you have for Bayes vs freq; and molecular vs biochem?

jppope 2y ago

These days I'm feeling like GenAi is basically an accuracy rate of 95% maybe 96%. Great at boilerplate, great at stuff you want an intern to do or maybe to outsource... but it really struggles with the valuable stuff. The errors are almost always in the most inconvenient places and they are hard to see... So I agree with Ben Evans on this one, what is one to do? the further you lean on it the worse your skills and specializations get. It is invaluable for some kinds of work greatly speeding you up, but then some of the things you would have caught take you down a rabbit hole that waste so much time. The tradeoffs here aren't great.

bakari500 2y ago
Yeah but you have 4 to 6 % error that’s not good even if you have dumb computer
daxfohl 2y ago
I think it's not the valuable stuff though. The valuable stuff is all the boilerplate, because, I don't want to do it. The rest, I actually have a stake in not only that it's done, but how it's done. And I'd rather be hands-on doing that and thinking about it as I do it. Having an AI do that isn't that valuable, and in fact robs me of the learning I acquire by doing it myself.
- deleted 2y ago
  [deleted]

smusamashah 2y ago

Watched recent Viva la dirt league videos on how trailers lie and do false promises. Now I see LLM as that marketing guy. Even if he knows everything, he can't help with lying. You can't trust anything he says no matter how authoritative he sounds, even if he is telling the truth you have know way of knowing. These deep research things are a waste of time if you can't trust the output. Code you can run and verify. How do you verify this.

deleted 2y ago
[deleted]

ano-ther 2y ago

Similar to what Derek Lowe found with a pharma example: https://www.science.org/content/blog-post/evaluation-deep-research-performance https://www.science.org/content/blog-post/evaluation-deep-re... > As with all LLM output, all of these things are presented in the same fluid, confident-sounding style: you have to know the material already to realize when your foot has gone through what was earlier solid flooring.

stogot 2y ago
Of which, people will surely die (when it is used to publish medical research by those wishing not to fail the publish-vs-perish situation)
- genewitch 2y ago
  I think the "delve" curve shows were already well into "AI papers" stage of civilization. It has probably been tamped down, now; the last thing j heard using it like a tell was notebookllm. Deep dive. "Yes, that's exactly right"

light_triad 2y ago

"Deep research" is super impressive, but so far is more "search the web and surf pages autonomously to aggregate relevant data" It is in many ways a workaround to Google's SEO poisoning. Doing very deep research requires a lot of context, cross-checking data, resourcefulness in sourcing and taste. Much of that context is industry specific and intuition plays a role in selecting different avenues to pursue and prioritisation. The error rates will go down but for the most difficult research it will be one tool among many rather than a replacement for the stack.

j_maffe 2y ago
> It is in many ways a workaround to Google's SEO poisoning. But the article goes into exactly how Deep Research fell exactly for the same SEO traps.
- light_triad 2y ago
  Right, hence the term "workaround" not "solution"

simonw 2y ago

When ChatGPT came out, one of the things we learned is that human society generally assumes a strong correlation between intelligence and the ability to string together grammatically correct sentences. As a result, many people assumed that even GPT-3.5 was wildly more "intelligent" than it actually was. I think Deep Research (and tools like it) offer an even stronger illustration of that same effect. Anything that can produce a well-formatted multiple page report with headings and citations surely must be of PhD-level intelligence, right? (Clearly not.)

SubiculumCode 2y ago
In some ways, it's a good tool to teach yourself to sus out the real clues to reliability, not format and authoritative tone.
- j_maffe 2y ago
  But that's the thing. The only way to truly find out if it's reliable (>90%) is to check the data yourself.
  - SubiculumCode 2y ago
    This is why metrics and leaderboards like these are so important (but under reported on): https://github.com/vectara/hallucination-leaderboard https://github.com/vectara/hallucination-leaderboard https://www.kaggle.com/facts-leaderboard https://www.kaggle.com/facts-leaderboard Google Gemni models seem to lead...hopefully the metrics aren't being gamed.
immibis 2y ago
That's been every LLM since GPT-2.
isaiahwp 2y ago
To be fair, OpenAI's the one marketing it as such.
jcgrillo 2y ago

deleted 2y ago

[deleted]

somerandomness 2y ago

Indeed, the main drawback of the various Deep Research implementation is the quality of sources is determined by SEO, which is often sketchy. Often the user has insight on what the right sources are and they may even be off-line on your computer. We built an alternative to do Deep Research (https://radpod.ai https://radpod.ai) on data you provide, instead of relying on Web results. We found this works a lot better in terms of quality of answers as the user can control the source quality.

deleted 2y ago
[deleted]

FeepingCreature 2y ago

> Are you telling me that today’s model gets this table 85% right and the next version will get it 85.5 or 91% correct? That doesn’t help me. If there are mistakes in the table, it doesn’t matter how many there are - I can’t trust it. If, on the other hand, you think that these models will go to being 100% right, that would change everything, but that would also be a binary change in the nature of these systems, not a percentage change, and we don’t know if that’s even possible. Of course, humans also make mistakes. There is a percentage, usually depending on the task but always below 100%, where the work is good enough to use, because that's how human labor works.

deleted 2y ago
[deleted]
semi-extrinsic 2y ago
If I'm paying a human, even a student working part-time or something, I expect "concrete facts extracted from the literature" to be correct at least 99.99% of the time. There is a huge gap from 85% to 99.99%.
- anthuswilliams 2y ago
  This does not at all resonate with my experience with human researchers, even highly paid ones. You still have to do a lot of work to verify their claims.
- FeepingCreature 2y ago
  Sure but there's no step change.
- nuancebydefault 2y ago
  You can expect that from a human, but if you don't know their reputation, you'd be lucky with the 85 percent. How do you even know if they understood the question correctly, used trusted sources, correct statistical models etc?
smusamashah 2y ago

jwpapi 2y ago

Yes the confidence tbh is getting a bit out of hand. I see the same thing with coding with our SAAS, once the problems get bigger I find myself more often than not starting to code the old way, even over "fixing ai’s code", because the issues are often too much. Ithink more certainty communication could help. Especially when they talk about docs or 3rd party packages etc. Regularly even Sonnet 3-7 just invents stuff...

skywhopper 2y ago

This article covers something early on that makes the question of “will models get to zero mistakes” pretty easy to answer: No. Even if they do the math right and find the data you ask for and never make any “facts” up, the sources of the data themselves carry a lot of context and connotation about how the data is gathered and what weight you can put on it. If anything, as LLMs become a more common way of ingesting the Internet, the sources of data themselves will start being SEOed to get chosen more often by the LLM purveyors. Add in paid sponsorship, and if anything, trust in the data from these sorts of Deep Research models will only get worse over time.

furyofantares 2y ago

I used deep research with o1-pro to try to fact/sanity check a current events thing a friend was talking about, read the results and followed the links it provided to get further info, and ended up on the verge of going down a rabbit hole that now looks more like a leftist(!) conspiracy theory.

furyofantares 2y ago
I didn't want to bring in specifics because I didn't feel like debating the specific thing, so I guess that made this post pretty hard to parse and I should have added more info. I was trying to convey that it had found some sources that, if I came across them naturally, I probably would have immediately recognized as fringe. The sources were threading together a number of true facts into a fringe narrative. The AI was able to get other sources on the true facts, but has no common sense, and I think ended up producing a MORE convincing presentation of the fringe theory than the source of the narrative. It sounded confident and used a number of extra sources to check facts even though the fringe narrative that threaded them all together was only from one site that you'd be somewhat apt to dismiss just by domain name if it was the only source you found.

deleted 2y ago

[deleted]

tippytippytango 2y ago

I think of it as the return to 10 blue links. It searches the web, finds stuff and summarizes it so I can decide which links to click. I ignore the narrative it constructs because it’s probably wrong. Which I forgive because it’s the hazmat suit for the internet I’ve always dreamed of. It gets in the trenches, braves the cookie popups, email signups, inline ads and overly clever web design so I don’t have to. That’s enough to forgive its attempts to create research narratives. But, I hope we can figure out a way to train a heaping spoonful of humility into future models.

CreepGin 2y ago
Agreed. Maybe we're moving toward a world where LLMs do all the searching, and "websites" just turn into data-only endpoints made for AI to consume. That'll have other big implications... Interesting times ahead.
- Seattle3503 2y ago
  What do the websites get out of that exchange?
  - fragmede 2y ago
    therein lies the problem, and is why Google search didn't disrupt itself until ChatGPT came around.
- ben_w 2y ago
  Interesting times, for sure. > and "websites" just turn into data-only endpoints made for AI to consume. As is already the case with humans, that only serves users to the extent that the websites' veracity is within the intelligence's ability to verify — all the problems we've had with blogspam etc. have been due to the subset of SEO where people abuse* the mechanisms to promote what they happen to be selling at the expense of genuine content. AI generated content is very good at seeming human, at seeming helpful. A "review website" which is all that, but with fake reviews that promote one brand over the others… a chain of such websites that link to each other to boost PageRank scores… which are then cross-linked with a huge number of social media bots… Will lead to a lot of people who think they're making an informed choice, but who were lied to about everything from their cornflakes to their president. * Tautologically, when it's not "abuse", SEO is only helping the search engine find the real content. I've seen places fail to perform any SEO including the legitimate kind.

spoaceman7777 2y ago

I, for one, have it in my prompt that GPT should end every message with a message about how sure it is about the answer, and a rating of "Extremely sure", "moderately sure", etc. It works surprisingly well. It also provides its reasoning about the quality of the sources, etc. (This is using GPT-4o of course, as it's the only mature GPT with web access) I highly recommend adding this to your default prompt.

afro88 2y ago
> It works surprisingly well What do you mean by this exactly? That it makes you feel better about what its said, or that its assessment of its answer is actually accurate?
- spoaceman7777 2y ago
  That's its assessment gives me a good picture of which ways to push with my next question, and which things to ask it to use its web search tool to find more information on. It's a conversation with AI, it's good to know its thought process on how certain it is of its conclusions, as it isn't infallible not is any human.

zeckalpha 2y ago

Two factors to consider: human performance and cost. Plenty of humans regularly make similar mistakes to the one in the Deep Research marketing, with more overhead than an LLM.

visarga 2y ago

The thig is, if you look at all the "Deep Research" benchmark scores, they never claim to be perfect. The problem was plain to see.

franze 2y ago

I am currently in India in a big city doing yoga for the first time as a westerner. I dont Google anything. Google maps, yeah. Google, no. Everything I want to know is much better answered by ChatGpt Deep Research. Ask a Question, Drink a Chai, Get a Great, Prioritised, structured Answer without spam or sifting through ad ridden SEO pages. It is a game changer, and at one point the will get rid of the "drink a chai" wait and it will kill the Google we know now.

nuancebydefault 2y ago

The problem with tools like deep research is that they imply good reasoning skills of the underlying technology. Artificial reasoning clearly exists, but is not refined enough to build these kind of technology on top of it. The reasoning is the fundamental of this system and all on top of it gets very unstable.

kgeist 2y ago

Deep Research, as it currently stands, is a jack of all trades but a master of none. Could this problem be mitigated by building domain-specific profiles curated by experts? For example, specifying which sources to prioritize, what to avoid, and so on. You would select a profile, and Deep Research would operate within its specific constraints, supplemented by expert knowledge.

Alifatisk 2y ago

What a beautiful website

localbolu 2y ago

> they don’t really have products either, just text boxes - and APIs for other people to build products. Isn't this a very valuable product in itself? Whatever happened to the phrase "When there is a gold rush, sell shovels"?

jcgrillo 2y ago

This is such embarrassing marketing from an organization (OpenAI) which presents itself to the world as a "research" entity.. They could have at least photoshopped the screenshot to make it look like it emitted correct information.

mirekrusin 2y ago

Not to take away from the main point of the article, which is true but: It seems to be at intern level according to the author - not bad if you ask me, no? Did he try to proceed as with an intern? ie. was it a dialogue? did he try to drop in this article into prompt and see what comes out? For skeptics my best advise is – do your usual work and at the end drop in your whole work with prompt to find issues etc. – it will be net positive result, I promise. And yes they do get better and it shouldn't get dismissed – the most fascinating part is precisely just that – not even their current state, but how fast they keep improving. One part which always bothers me a bit with this type of arguments – why on earth are we assuming that human does it 100% correctly? Aren't humans also making similar mistakes? IMHO there is some similarity with young geniuses – they get tons of stuff right and it's impressive however total, unexpected failures occur which feel weird – in my opinion it's a matter of focused training similar to how you'd teach a genius. It's worth taking step back and recognizing in how many diverse contexts we're using (like now, today, not in 5 years) models like grok3 or claude3.7 – the goalpost seem to have moved to "beyond any human expert on all subjects".

marcuschong 2y ago

"I can say that these systems are amazing, but get things wrong all the time in ways that matter, and so the best uses cases so far are things where the error rate doesn’t matter or where it’s easy to see." That's probably how we should all be using LLMs.