7 ms·

>Now, everyone basically has a personal TA, ready to go at all hours of the day This simply hasn't been my experience. Its too shallow. The deeper I go, the l

>Now, everyone basically has a personal TA, ready to go at all hours of the day This simply hasn't been my experience. Its too shallow. The deeper I go, the less it seems to be useful. This happens quick for me. Also, god forbid you're researching a complex and possibly controversial subject and you want it to find reputable sources or particularly academic ones.

kianN 1y ago
I built a public tool a while back for some of my friends in grad school to support this sort of deep academic research use case. Sharing in case it is helpful: https://sturdystatistics.com/deepdive?search_type=external&q=https://www.semanticscholar.org/paper/6052486bc9144dc1730c12bf35323af3792a1fd0&engine=cn_all https://sturdystatistics.com/deepdive?search_type=external&q...
ACCount36 1y ago
It is shallow. But as long as what you're asking it of is the kind of material covered in high school or college, it's fairly reliable. This generation of AI doesn't yet have the knowledge depth of a seasoned university professor. It's the kind of teacher that you should, eventually, surpass.
scarmig 1y ago
I've found it excels at some things: 1) The broad overview of a topic 2) When I have a vague idea, it helps me narrow down the correct terminology for it 3) Providing examples of a particular category ("are there any examples of where v1 in the visual cortex develops in a disordered way?") 4) "Tell me the canonical textbooks in field X" 5) Posing math exercises 6) Free form branching--while talking about one topic, I want to shift to another that is distinct but related. I agree they leave a lot to be desired when digging very deeply into a topic. And my biggest pet peeve is when they hallucinate fake references ("tell me papers that investigate this topic" will, for any sufficiently obscure topic, result in a bunch of very promising paper titles that are wholely invented).
- CJefferson 1y ago
  These things are moving so quickly, but I teach a 2nd year combinatorics course, and about 3 months ago I tried th latest chatGPT and Deepseek -- they could answer very standard questions, but were wrong for more advanced questions, but often in quite subtle ways. I actually set a piece of homework "marking" chatGPT, which went well and students seemed to enjoy!
  - teaearlgraycold 1y ago
    That’s a great idea to both teach the subject and AI skepticism.
  - scarmig 1y ago
    Very clever and approachable, and I've been unintentionally giving myself that exercise for awhile now. Who knows how long it will remain viable, though.
  - jennyholzer 1y ago
    that's a cool assignment!
  - p1esk 1y ago
    When you say the latest chatGPT, do you mean o3?
    - CJefferson 1y ago
      Whatever was best on a paid account 3 months ago. I was quite disappointed to be honest, based on what I had been hearing.
      - p1esk 1y ago
        I think by default ChatGPT will choose 4o for you. So unless you actually chose o3 you haven’t used the best model.
  - Julien_r2 1y ago
SLWW 1y ago
My core problem with LLMs is as you say; it's good for some simpler concepts, tasks, etc. but when you need to dive into more complex topics it will oversimplify, give you what you didn't ask for, or straight up lie by omission. History is a great example, if you ask an LLM about a vaguely difficult period in history it will just give you one side and act like the other doesn't exist, or if there is another side, it will paint them in a very negative light which often is poorly substantiated; people don't just wake up and decide one day to be irrationally evil with no reason, if you believe that then you are a fool... although LLMs would agree with you more times than not since it's convenient. The result of these things is a form of gatekeeping, give it a few years and basic knowledge will be almost impossible to find if it is deemed "not useful" whether that's an outdated technology that the LLM doesn't seem talked about very much anymore or a ideological issue that doesn't fall in line with TOS or common consensus.
- pengstrom 1y ago
  The part about history perspectives sounds interesting. I haven't noticed this. Please post any concrete/specific examples you've encountered!
  - SLWW 1y ago
    - Rhodesia (lock step with the racial-first reasoning, underplays Britain's failures to support that which they helped establish; makes the colonists look hateful when they were dealing with terrorists which the British supported) - Bombing of Dresden, death stats as well as how long the bombing went on for (Arthur Harris is considered a war-criminal to this day for that; LLLMs highlight easily falsifiable claims by Nazi's to justify low estimates without providing much in the way of verifiable claims outside of a select few, questionable, sources. If the low-estimate is to be believed, then it seems absurd that Harris would be considered a war-criminal in light of what crimes we allow today in warfare) - Ask it about the Crusades, often if forgets the sacking of St. Peter's in Rome around 846 AD, usually painting the Papacy as a needlessly hateful and violent people during that specific Crusade. Which was horrible, bloody as well as immensely destructive (I don't defend the Crusades), but paints the Islamic forces as victims, which they were eventually, but not at the beginning, at the beginning they were the aggressors bent on invading Rome. - Ask it about the Six-Day War (1967) and contrast that with several different sources on both sides and you'll see a different portrayal even by those who supported the actions taken. These are just the four that come to my memory at this time. Most LLMs seem cagey about these topics; I believe this is due to an accepted notion that anything that could "justify" hatred or dislike of a people group or class that is in favor -- according to modern politics -- will be classified as hateful rhetoric, which is then omitted from the record. The issue lies in the fact that to understand history, we need to understand what happened, not how it is perceived, politically, after the fact. History helps inform us about the issues of today, and it is important, above all other agendas, to represent the truth of history, keeping an accurate account (or simply allowing others to read differing accounts without heavy bias). LLMs are restricted in this way quite egregiously; "those who do not study history are doomed to repeat it", but if this continues, no one will have the ability to know history and are therefore forced to repeat it.
Xenoamorphous 1y ago
Can you share some examples?
- no_wizard 1y ago
  Try doing deep research on the Israel - Palestine relations. That’s a good baseline. You’ll find it starts spitting out really useless stuff fast, or will try to give sources that don’t exist or are not reputable.
tsumnia 1y ago
IT can be beneficial for making your initial assessment, but you'll need to dig deeper for something meaningful. For example, I recently used Gemini's Deep Research to do some literature review on educational Color Theory in relation to PowerPoint presentations [1]. I know both areas rather well, but I wanted to have some links between the two for some research that I am currently doing. I'd say that companies like Google and OpenAI are aware of the "reputable" concerns the Internet is expressing and addressing them. This tech is going to be, if not already is, very powerful for education. [1] http://bit.ly/4mc4UHG http://bit.ly/4mc4UHG
- fakedang 1y ago
  Taking a Gemini Deep Research output and feeding it to NotebookLM to create audio overviews is my current podcast go-to. Sometimes I do a quick Google and add in a few detailed but overly verbose documents or long form YouTube videos, and the result is better than 99% of the podcasts out there, including those by some academics.
  - hammyhavoc 1y ago
    No wonder there are so many confident people spouting total rubbish on technical forums.
EGreg 1y ago
Try to red team blue team with it Blue team you throw out concepts and have it steelman them Red team you can literally throw any kind of stress test at your idea Alternate like this and you will learn A great prompt is “give me the top 10 xyz things” and then you can explore Back when I was in 2006 I used Wikipedia to prepare for job interviews :)
HPsquared 1y ago
Human interlocutors have similar issues.
beambot 1y ago
The worst is when it's confidently wrong about things... Thankfully, this occurance is becoming less & less common -- or at least, it's boundary is beyond my subject matter expertise.
Teever 1y ago
It sounds like it is a good tool for getting you up to speed on a subject and you can leverage that newfound familiarity to better search for reputable sources on existing platforms like google scholar or arXiv.
jjfoooo4 1y ago
It's a floor raiser, not a ceiling raiser. It helps you get up to speed on general conventions and consensus on a topic, less so on going deep on controversial or highly specialized topics
- deleted 1y ago
  [deleted]
- felipehummel 1y ago
  That's the best and succinct description of using ChatGPT for this kind of things: it's a floor raiser, not a ceiling raiser.
  - jjfoooo4 1y ago
    If it's of interest, I expanded on this in a blog post: https://elroy.bot/blog/2025/07/29/ai-is-a-floor-raiser-not-a-ceiling-raiser.html https://elroy.bot/blog/2025/07/29/ai-is-a-floor-raiser-not-a...
neutronicus 1y ago
Hmm. I have had pretty productive conversations with ChatGPT about non-linear optimization. Granted, that's probably well-trodden ground, to which model developers are primed to pay attention, and I'm (a) a relative novice with (b) very strong math skills from another domain (computational physics). So Chuck and I are probably both set up for success.
II2II 1y ago
> Also, god forbid you're researching a complex and possibly controversial subject and you want it to find reputable sources or particularly academic ones. That's fine. Recognize the limits of LLMs and don't use them in those cases. Yet that is something you should be doing regardless of the source. There are plenty of non-reputable sources in academic libraries and there are plenty of non-reputable sources from professionals in any given field. That is particularly true when dealing with controversial topics or historical sources.
marcosdumay 1y ago
> and you want it to find reputable sources Ask it for sources. The two things where LLMs excel is by filling the sources on some claim you give it (lots will be made up, but there isn't anything better out there) and by giving you queries you can search for some description you give it.
- chrisweekly 1y ago
  Also, Perplexity.ai cites its sources by default.
- golly_ned 1y ago
  It often invents sources. At least for me.
jlebar 1y ago
> Its too shallow. The deeper I go, the less it seems to be useful. This happens quick for me. You must be using a free model like GPT-4o (or the equivalent from another provider)? I find that o3 is consistently able to go deeper than me in anything I'm a nonexpert in, and usually can keep up with me in those areas where I am an expert. If that's not the case for you I'd be very curious to see a full conversation transcript (in chatgpt you can share these directly from the UI).
- no_wizard 1y ago
  I have access to the highest tier paid versions of ChatGPT and Google Gemini, I've tried different models, tuning things like size of context windows etc. I know it has nothing to do with this. I simply hit a wall eventually. I unfortunately am not at liberty to share the chats though. They're work related (I very recently ended up at a place where we do thorny research). A simple one though, is researching Israel - Palestine relations since 1948. It starts off okay (usually) but it goes off the rails eventually with bad sourcing, fictitious sourcing, and/or hallucinations. Sometimes I actually hit a wall where it repeats itself over and over and I suspect its because the information is simply not captured by the model. FWIW, if these models had live & historic access to Reuters and Bloomberg terminals I think they might be better at a range of tasks I find them inadequate for, maybe.
  - johnisgood 1y ago
    > I unfortunately am not at liberty to share the chats though. I have bad news for you. If you shared it with ChatGPT (which you most likely did), then whatever it is that you are trying to keep hidden or private, is not actually hidden or private anymore, it is stored on their servers, and most likely will be trained on that chat. Use local models instead in such cases.
vonneumannstan 1y ago
>Its too shallow. The deeper I go, the less it seems to be useful. This happens quick for me. If its a subject you are just learning how can you possibly evaluate this?
- neutronicus 1y ago
  If you're a math-y person trying to get up to speed in some other math-y field you can discern useless LLM output pretty quickly even as a relative novice. Falling apart under pointed questioning, saying obviously false things, etc.
- Sharlin 1y ago
  It's easy to recognize that something is wrong if it's wrong enough.
epolanski 1y ago
I really think that 90% of such comments come from a lack of knowledge on how to use LLMs for research. It's not a criticism, the landscape moves fast and it takes time to master and personalize a flow to use an LLM as a research assistant. Start with something such as NotebookLM.
- no_wizard 1y ago
  I use them and stay up to date reasonably. I have used NotebookLM, I have access to advanced models through my employer and personally, and I have done alot of research on LLMs and using them effectively. They simply have limitations, especially on deep pointed subject matters where you want depth not breadth, and honestly I'm not sure why these limitations exist but I'm not working directly on these systems. Talk to Gemini or ChatGPT about mental health things, thats a good example of what I'm talking about. As recently as two weeks ago my colleagues found that even when heavily tuned, they still managed to become 'pro suicide' if given certain lines of questioning.
- johnnyanmac 1y ago
  And if we assume this is a knowledgable, technical community: how do you feel about the general populaces ability to use LLM's for research, without the skepticism needed to correct it?
kenjackson 1y ago
“The deeper I go, the less it seems to be useful. This happens quick for me. Also, god forbid you're researching a complex and possibly controversial subject and you want it to find reputable sources or particularly academic ones.” These things also apply to humans. A year or so ago I thought I’d finally learn more about the Israeli/Palestinians conflict. Turns out literally every source that was recommended to me by some reputable source was considered completely non-credible by another reputable one. That said I’ve found ChatGPT to be quite good at math and programming and I can go pretty deep at both. I can definitely trip it into mistakes (eg it seems to use calculations to “intuit” its way around sometimes and you can find dev cases where the calls will lead it the wrong directions), but I also know enough to know how to keep it on rails.
- Liftyee 1y ago
  Re: conflicts and politics etc. I've anecdotally found that real world things like these tend to be nuanced, and that sources (especially on the internet) are disincentivised in various ways from actually showing nuance. This leads to "side-taking" and a lack of "middle-ground" nuanced sources, when the reality lies somewhere in the middle. Might be linked to the phenomenon where in an environment where people "take sides", those who display moderate opinions are simply ostracized by both sides. Curious to hear people's thoughts and disagreements on this.
  - wahern 1y ago
    I think the Israeli/Palestinian conflict is an example where studying the history is in some sense counter-productive. There's more than a century of atrocities that justify each subsequent reaction; the veritable cycle of violence. And whichever atrocity grabs you first (partly based on present cultural narratives) will color how you perceive everything else. Moreover, the conflict is unfolding. What matters isn't what happened 100 years ago, or even 50 years ago, but what has happened recently and is happening. A neighbor of mine who recently passed was raised in Israel. Born circa 1946 (there's black & white footage of her as a baby aboard, IIRC, the ship Exodus 1947), she has vivid memories as a child of Palestinian Imams calling out from the mosques to "kill the Jews". She was a beautiful, kind soul who, for example, freely taught adult education to immigrants (of all sorts), but who one time admitted to me that she utterly despised Arabs. That's all you need to know, right there, to understand why Israel is doing what it's doing. Not so much what happened in the past to make people feel that way, but that many Israelis actually, viscerally feel this way today, justifiably or not but in any event rooted in memories and experiences seared into their conscience. Suffice it to say, most Palestinians have similar stories and sentiments of their own, one of the expressions of which was seen on October 7th. And yet at the same time, after the first few months of the Gaza War she was so disgusted that she said she wanted to renounce her Israeli citizenship. (I don't know how sincere she was in saying this; she died not long after.) And, again, that's all you need to know to see how the conflict can be resolved, if at all; not by understanding and reconciling the history, but merely choosing to stop justifying the violence and moving forward. How the collective action problem might be resolved, within Israeli and Palestinian societies and between them... that's a whole 'nother dilemma. Using AI/ML to study history is interesting in that it even further removes one from actual human experience. Hearing first hand accounts, even if anecdotal, conveys information you can't acquire from a book; reading a book conveys information and perspective you can't get from a shorter work, like a paper or article; and AI/ML summaries elide and obscure yet more substance.
prats226 1y ago
Can you give a specific example where at certain depth it has stopped becoming useful?
terabyterex 1y ago
This can happen if you use the free model and not a paid deep research model. You can use a gpt model and ask things like , "how many moons does Jupiter have?" But if you want to ask, "can you go on the web a research the affects that chamical a has had on our water supply a cite sources?", you will need to use a deep research model.
- hammyhavoc 1y ago
  Why not do the research yourself rather than risk it misinterpreting? I FAFO'd repeatedly with that, and it is just horribly unreliable.
jasondigitized 1y ago
If we have custom trained LLMs per subject doesn't that solve the problem. The shallow problem seems really easy to solve
gojomo 1y ago
Grandparent testimony of success, & parent testimony of frustration, are both just wispy random gossip when they don't specify which LLMs delivered the reported experiences. The quality varies wildly across models & versions. With humans, the statement "my tutor was great" and "my tutor was awful" reflect very little on "tutoring" in general, and are barely even responses to each other withou more specificity about the quality of tutor involved. Same with AI models.
- no_wizard 1y ago
  Latest OpenAI, Latest Gemini models, also tried with latest LLAMA but I didn’t expect much there. I have no access to anthropic right now to compare that. It’s an ongoing problem in my experience
waynesonfire 1y ago
It's not a doctoral adviser.
CamperBob2 1y ago
What is "it"? Be specific: are you using some obsolete and/or free model? What specific prompt(s) convinced you that there was no way forward?
melenaboija 1y ago
I validate models in finance, and this is by far the best tool created for that purpose. I'd compare financial model validation to a Master's level task, where you're working with well established concepts, but at a deep, technical level. LLMs excel at that: ithey understand model assumptions, know what needs to be tested to ensure correctness, and can generate the necessary code and calculations to perform those tests. And finally, they can write the reports. Model Validation groups are one of the targets for LLMs.
- no_wizard 1y ago
  That’s one aspect of quantitative finance, and I agree. Elsewhere I noted that anything that is structured data + computation adjacent it has an easier time with, even excels in many cases. It doesn’t cover the other aspects of finance, perhaps may be considered advanced (to a regular person at least) but less quantitative. Try having it reason out a “cigar butt” strategy and see if returns anything useful about companies that fit the mold from a prepared source. Granted this isn’t quant finance modeling, but it’s a relatively easy thing as a human to do, and I didn’t find LLMs up to the task
EchoReflection 1y ago
I have found that being very specific and asking things like "can you tell me what another perspective might be, such that I can understand potential counter-arguments might be, and how people with other views might see this topic?" can be helpful when dealing with complex/nuanced/contentious subjects. Likewise with regard to "reputable" sources.
noosphr 1y ago
This is where feeding in extra context matters. Paste in text that shows up from a google search, textbooks preferred, to get in depth answers. No one builds multi shot search tools because they eat tokens like no ones business, but I've deployed them internal to a company with rave reviews at the cost of $200 per seat per day.
karaterobot 1y ago
What are some subjects that ChatGPT has given only shallow instruction on? I'll tell you that I recently found it the best resource on the web for teaching me about the 30 Years War. I was reading a collection of primary source documents, and was able to interview ChatGPT about them. Last week I used it to learn how to create and use Lehmer codes, and its explanation was perfect, and much easier to understand than, for example, Wikipedia. I ask it about truck repair stuff all the time, and it is also great at that. I don't think it's great at literary analysis, but for factual stuff it has only ever blown away my expectations at how useful it is.
dankwizard 1y ago
This is the part where you actually need to think and wonder if AI is the right tool in this particular purpose. Unfortunately you can't completely turn your brain off just yet.
wyager 1y ago
> god forbid you're researching a complex and possibly controversial subject and you want it to find reputable sources If you're really researching something complex/controversial, there may not be any

kianN 1y ago

I built a public tool a while back for some of my friends in grad school to support this sort of deep academic research use case. Sharing in case it is helpful: https://sturdystatistics.com/deepdive?search_type=external&q=https://www.semanticscholar.org/paper/6052486bc9144dc1730c12bf35323af3792a1fd0&engine=cn_all https://sturdystatistics.com/deepdive?search_type=external&q...

ACCount36 1y ago

It is shallow. But as long as what you're asking it of is the kind of material covered in high school or college, it's fairly reliable. This generation of AI doesn't yet have the knowledge depth of a seasoned university professor. It's the kind of teacher that you should, eventually, surpass.

scarmig 1y ago

I've found it excels at some things: 1) The broad overview of a topic 2) When I have a vague idea, it helps me narrow down the correct terminology for it 3) Providing examples of a particular category ("are there any examples of where v1 in the visual cortex develops in a disordered way?") 4) "Tell me the canonical textbooks in field X" 5) Posing math exercises 6) Free form branching--while talking about one topic, I want to shift to another that is distinct but related. I agree they leave a lot to be desired when digging very deeply into a topic. And my biggest pet peeve is when they hallucinate fake references ("tell me papers that investigate this topic" will, for any sufficiently obscure topic, result in a bunch of very promising paper titles that are wholely invented).

CJefferson 1y ago
These things are moving so quickly, but I teach a 2nd year combinatorics course, and about 3 months ago I tried th latest chatGPT and Deepseek -- they could answer very standard questions, but were wrong for more advanced questions, but often in quite subtle ways. I actually set a piece of homework "marking" chatGPT, which went well and students seemed to enjoy!
- teaearlgraycold 1y ago
  That’s a great idea to both teach the subject and AI skepticism.
- scarmig 1y ago
  Very clever and approachable, and I've been unintentionally giving myself that exercise for awhile now. Who knows how long it will remain viable, though.
- jennyholzer 1y ago
  that's a cool assignment!
- p1esk 1y ago
  When you say the latest chatGPT, do you mean o3?
  - CJefferson 1y ago
    Whatever was best on a paid account 3 months ago. I was quite disappointed to be honest, based on what I had been hearing.
    - p1esk 1y ago
      I think by default ChatGPT will choose 4o for you. So unless you actually chose o3 you haven’t used the best model.
- Julien_r2 1y ago

SLWW 1y ago

My core problem with LLMs is as you say; it's good for some simpler concepts, tasks, etc. but when you need to dive into more complex topics it will oversimplify, give you what you didn't ask for, or straight up lie by omission. History is a great example, if you ask an LLM about a vaguely difficult period in history it will just give you one side and act like the other doesn't exist, or if there is another side, it will paint them in a very negative light which often is poorly substantiated; people don't just wake up and decide one day to be irrationally evil with no reason, if you believe that then you are a fool... although LLMs would agree with you more times than not since it's convenient. The result of these things is a form of gatekeeping, give it a few years and basic knowledge will be almost impossible to find if it is deemed "not useful" whether that's an outdated technology that the LLM doesn't seem talked about very much anymore or a ideological issue that doesn't fall in line with TOS or common consensus.

pengstrom 1y ago
The part about history perspectives sounds interesting. I haven't noticed this. Please post any concrete/specific examples you've encountered!
- SLWW 1y ago
  - Rhodesia (lock step with the racial-first reasoning, underplays Britain's failures to support that which they helped establish; makes the colonists look hateful when they were dealing with terrorists which the British supported) - Bombing of Dresden, death stats as well as how long the bombing went on for (Arthur Harris is considered a war-criminal to this day for that; LLLMs highlight easily falsifiable claims by Nazi's to justify low estimates without providing much in the way of verifiable claims outside of a select few, questionable, sources. If the low-estimate is to be believed, then it seems absurd that Harris would be considered a war-criminal in light of what crimes we allow today in warfare) - Ask it about the Crusades, often if forgets the sacking of St. Peter's in Rome around 846 AD, usually painting the Papacy as a needlessly hateful and violent people during that specific Crusade. Which was horrible, bloody as well as immensely destructive (I don't defend the Crusades), but paints the Islamic forces as victims, which they were eventually, but not at the beginning, at the beginning they were the aggressors bent on invading Rome. - Ask it about the Six-Day War (1967) and contrast that with several different sources on both sides and you'll see a different portrayal even by those who supported the actions taken. These are just the four that come to my memory at this time. Most LLMs seem cagey about these topics; I believe this is due to an accepted notion that anything that could "justify" hatred or dislike of a people group or class that is in favor -- according to modern politics -- will be classified as hateful rhetoric, which is then omitted from the record. The issue lies in the fact that to understand history, we need to understand what happened, not how it is perceived, politically, after the fact. History helps inform us about the issues of today, and it is important, above all other agendas, to represent the truth of history, keeping an accurate account (or simply allowing others to read differing accounts without heavy bias). LLMs are restricted in this way quite egregiously; "those who do not study history are doomed to repeat it", but if this continues, no one will have the ability to know history and are therefore forced to repeat it.

Xenoamorphous 1y ago

Can you share some examples?

no_wizard 1y ago
Try doing deep research on the Israel - Palestine relations. That’s a good baseline. You’ll find it starts spitting out really useless stuff fast, or will try to give sources that don’t exist or are not reputable.

tsumnia 1y ago

IT can be beneficial for making your initial assessment, but you'll need to dig deeper for something meaningful. For example, I recently used Gemini's Deep Research to do some literature review on educational Color Theory in relation to PowerPoint presentations [1]. I know both areas rather well, but I wanted to have some links between the two for some research that I am currently doing. I'd say that companies like Google and OpenAI are aware of the "reputable" concerns the Internet is expressing and addressing them. This tech is going to be, if not already is, very powerful for education. [1] http://bit.ly/4mc4UHG http://bit.ly/4mc4UHG

fakedang 1y ago
Taking a Gemini Deep Research output and feeding it to NotebookLM to create audio overviews is my current podcast go-to. Sometimes I do a quick Google and add in a few detailed but overly verbose documents or long form YouTube videos, and the result is better than 99% of the podcasts out there, including those by some academics.
- hammyhavoc 1y ago
  No wonder there are so many confident people spouting total rubbish on technical forums.

EGreg 1y ago

Try to red team blue team with it Blue team you throw out concepts and have it steelman them Red team you can literally throw any kind of stress test at your idea Alternate like this and you will learn A great prompt is “give me the top 10 xyz things” and then you can explore Back when I was in 2006 I used Wikipedia to prepare for job interviews :)

HPsquared 1y ago

Human interlocutors have similar issues.

beambot 1y ago

The worst is when it's confidently wrong about things... Thankfully, this occurance is becoming less & less common -- or at least, it's boundary is beyond my subject matter expertise.

Teever 1y ago

It sounds like it is a good tool for getting you up to speed on a subject and you can leverage that newfound familiarity to better search for reputable sources on existing platforms like google scholar or arXiv.

jjfoooo4 1y ago

It's a floor raiser, not a ceiling raiser. It helps you get up to speed on general conventions and consensus on a topic, less so on going deep on controversial or highly specialized topics

deleted 1y ago
[deleted]
felipehummel 1y ago
That's the best and succinct description of using ChatGPT for this kind of things: it's a floor raiser, not a ceiling raiser.
- jjfoooo4 1y ago
  If it's of interest, I expanded on this in a blog post: https://elroy.bot/blog/2025/07/29/ai-is-a-floor-raiser-not-a-ceiling-raiser.html https://elroy.bot/blog/2025/07/29/ai-is-a-floor-raiser-not-a...

neutronicus 1y ago

Hmm. I have had pretty productive conversations with ChatGPT about non-linear optimization. Granted, that's probably well-trodden ground, to which model developers are primed to pay attention, and I'm (a) a relative novice with (b) very strong math skills from another domain (computational physics). So Chuck and I are probably both set up for success.

II2II 1y ago

> Also, god forbid you're researching a complex and possibly controversial subject and you want it to find reputable sources or particularly academic ones. That's fine. Recognize the limits of LLMs and don't use them in those cases. Yet that is something you should be doing regardless of the source. There are plenty of non-reputable sources in academic libraries and there are plenty of non-reputable sources from professionals in any given field. That is particularly true when dealing with controversial topics or historical sources.

marcosdumay 1y ago

> and you want it to find reputable sources Ask it for sources. The two things where LLMs excel is by filling the sources on some claim you give it (lots will be made up, but there isn't anything better out there) and by giving you queries you can search for some description you give it.

chrisweekly 1y ago
Also, Perplexity.ai cites its sources by default.
golly_ned 1y ago
It often invents sources. At least for me.

jlebar 1y ago

> Its too shallow. The deeper I go, the less it seems to be useful. This happens quick for me. You must be using a free model like GPT-4o (or the equivalent from another provider)? I find that o3 is consistently able to go deeper than me in anything I'm a nonexpert in, and usually can keep up with me in those areas where I am an expert. If that's not the case for you I'd be very curious to see a full conversation transcript (in chatgpt you can share these directly from the UI).

no_wizard 1y ago
I have access to the highest tier paid versions of ChatGPT and Google Gemini, I've tried different models, tuning things like size of context windows etc. I know it has nothing to do with this. I simply hit a wall eventually. I unfortunately am not at liberty to share the chats though. They're work related (I very recently ended up at a place where we do thorny research). A simple one though, is researching Israel - Palestine relations since 1948. It starts off okay (usually) but it goes off the rails eventually with bad sourcing, fictitious sourcing, and/or hallucinations. Sometimes I actually hit a wall where it repeats itself over and over and I suspect its because the information is simply not captured by the model. FWIW, if these models had live & historic access to Reuters and Bloomberg terminals I think they might be better at a range of tasks I find them inadequate for, maybe.
- johnisgood 1y ago
  > I unfortunately am not at liberty to share the chats though. I have bad news for you. If you shared it with ChatGPT (which you most likely did), then whatever it is that you are trying to keep hidden or private, is not actually hidden or private anymore, it is stored on their servers, and most likely will be trained on that chat. Use local models instead in such cases.

vonneumannstan 1y ago

>Its too shallow. The deeper I go, the less it seems to be useful. This happens quick for me. If its a subject you are just learning how can you possibly evaluate this?

neutronicus 1y ago
If you're a math-y person trying to get up to speed in some other math-y field you can discern useless LLM output pretty quickly even as a relative novice. Falling apart under pointed questioning, saying obviously false things, etc.
Sharlin 1y ago
It's easy to recognize that something is wrong if it's wrong enough.

epolanski 1y ago

I really think that 90% of such comments come from a lack of knowledge on how to use LLMs for research. It's not a criticism, the landscape moves fast and it takes time to master and personalize a flow to use an LLM as a research assistant. Start with something such as NotebookLM.

no_wizard 1y ago
I use them and stay up to date reasonably. I have used NotebookLM, I have access to advanced models through my employer and personally, and I have done alot of research on LLMs and using them effectively. They simply have limitations, especially on deep pointed subject matters where you want depth not breadth, and honestly I'm not sure why these limitations exist but I'm not working directly on these systems. Talk to Gemini or ChatGPT about mental health things, thats a good example of what I'm talking about. As recently as two weeks ago my colleagues found that even when heavily tuned, they still managed to become 'pro suicide' if given certain lines of questioning.
johnnyanmac 1y ago
And if we assume this is a knowledgable, technical community: how do you feel about the general populaces ability to use LLM's for research, without the skepticism needed to correct it?

kenjackson 1y ago

“The deeper I go, the less it seems to be useful. This happens quick for me. Also, god forbid you're researching a complex and possibly controversial subject and you want it to find reputable sources or particularly academic ones.” These things also apply to humans. A year or so ago I thought I’d finally learn more about the Israeli/Palestinians conflict. Turns out literally every source that was recommended to me by some reputable source was considered completely non-credible by another reputable one. That said I’ve found ChatGPT to be quite good at math and programming and I can go pretty deep at both. I can definitely trip it into mistakes (eg it seems to use calculations to “intuit” its way around sometimes and you can find dev cases where the calls will lead it the wrong directions), but I also know enough to know how to keep it on rails.

Liftyee 1y ago
Re: conflicts and politics etc. I've anecdotally found that real world things like these tend to be nuanced, and that sources (especially on the internet) are disincentivised in various ways from actually showing nuance. This leads to "side-taking" and a lack of "middle-ground" nuanced sources, when the reality lies somewhere in the middle. Might be linked to the phenomenon where in an environment where people "take sides", those who display moderate opinions are simply ostracized by both sides. Curious to hear people's thoughts and disagreements on this.
- wahern 1y ago
  I think the Israeli/Palestinian conflict is an example where studying the history is in some sense counter-productive. There's more than a century of atrocities that justify each subsequent reaction; the veritable cycle of violence. And whichever atrocity grabs you first (partly based on present cultural narratives) will color how you perceive everything else. Moreover, the conflict is unfolding. What matters isn't what happened 100 years ago, or even 50 years ago, but what has happened recently and is happening. A neighbor of mine who recently passed was raised in Israel. Born circa 1946 (there's black & white footage of her as a baby aboard, IIRC, the ship Exodus 1947), she has vivid memories as a child of Palestinian Imams calling out from the mosques to "kill the Jews". She was a beautiful, kind soul who, for example, freely taught adult education to immigrants (of all sorts), but who one time admitted to me that she utterly despised Arabs. That's all you need to know, right there, to understand why Israel is doing what it's doing. Not so much what happened in the past to make people feel that way, but that many Israelis actually, viscerally feel this way today, justifiably or not but in any event rooted in memories and experiences seared into their conscience. Suffice it to say, most Palestinians have similar stories and sentiments of their own, one of the expressions of which was seen on October 7th. And yet at the same time, after the first few months of the Gaza War she was so disgusted that she said she wanted to renounce her Israeli citizenship. (I don't know how sincere she was in saying this; she died not long after.) And, again, that's all you need to know to see how the conflict can be resolved, if at all; not by understanding and reconciling the history, but merely choosing to stop justifying the violence and moving forward. How the collective action problem might be resolved, within Israeli and Palestinian societies and between them... that's a whole 'nother dilemma. Using AI/ML to study history is interesting in that it even further removes one from actual human experience. Hearing first hand accounts, even if anecdotal, conveys information you can't acquire from a book; reading a book conveys information and perspective you can't get from a shorter work, like a paper or article; and AI/ML summaries elide and obscure yet more substance.

prats226 1y ago

Can you give a specific example where at certain depth it has stopped becoming useful?

terabyterex 1y ago

This can happen if you use the free model and not a paid deep research model. You can use a gpt model and ask things like , "how many moons does Jupiter have?" But if you want to ask, "can you go on the web a research the affects that chamical a has had on our water supply a cite sources?", you will need to use a deep research model.

hammyhavoc 1y ago
Why not do the research yourself rather than risk it misinterpreting? I FAFO'd repeatedly with that, and it is just horribly unreliable.

jasondigitized 1y ago

If we have custom trained LLMs per subject doesn't that solve the problem. The shallow problem seems really easy to solve

gojomo 1y ago

Grandparent testimony of success, & parent testimony of frustration, are both just wispy random gossip when they don't specify which LLMs delivered the reported experiences. The quality varies wildly across models & versions. With humans, the statement "my tutor was great" and "my tutor was awful" reflect very little on "tutoring" in general, and are barely even responses to each other withou more specificity about the quality of tutor involved. Same with AI models.

no_wizard 1y ago
Latest OpenAI, Latest Gemini models, also tried with latest LLAMA but I didn’t expect much there. I have no access to anthropic right now to compare that. It’s an ongoing problem in my experience

waynesonfire 1y ago

It's not a doctoral adviser.

CamperBob2 1y ago

What is "it"? Be specific: are you using some obsolete and/or free model? What specific prompt(s) convinced you that there was no way forward?

melenaboija 1y ago

I validate models in finance, and this is by far the best tool created for that purpose. I'd compare financial model validation to a Master's level task, where you're working with well established concepts, but at a deep, technical level. LLMs excel at that: ithey understand model assumptions, know what needs to be tested to ensure correctness, and can generate the necessary code and calculations to perform those tests. And finally, they can write the reports. Model Validation groups are one of the targets for LLMs.

no_wizard 1y ago
That’s one aspect of quantitative finance, and I agree. Elsewhere I noted that anything that is structured data + computation adjacent it has an easier time with, even excels in many cases. It doesn’t cover the other aspects of finance, perhaps may be considered advanced (to a regular person at least) but less quantitative. Try having it reason out a “cigar butt” strategy and see if returns anything useful about companies that fit the mold from a prepared source. Granted this isn’t quant finance modeling, but it’s a relatively easy thing as a human to do, and I didn’t find LLMs up to the task

EchoReflection 1y ago

I have found that being very specific and asking things like "can you tell me what another perspective might be, such that I can understand potential counter-arguments might be, and how people with other views might see this topic?" can be helpful when dealing with complex/nuanced/contentious subjects. Likewise with regard to "reputable" sources.

noosphr 1y ago

This is where feeding in extra context matters. Paste in text that shows up from a google search, textbooks preferred, to get in depth answers. No one builds multi shot search tools because they eat tokens like no ones business, but I've deployed them internal to a company with rave reviews at the cost of $200 per seat per day.

karaterobot 1y ago

What are some subjects that ChatGPT has given only shallow instruction on? I'll tell you that I recently found it the best resource on the web for teaching me about the 30 Years War. I was reading a collection of primary source documents, and was able to interview ChatGPT about them. Last week I used it to learn how to create and use Lehmer codes, and its explanation was perfect, and much easier to understand than, for example, Wikipedia. I ask it about truck repair stuff all the time, and it is also great at that. I don't think it's great at literary analysis, but for factual stuff it has only ever blown away my expectations at how useful it is.

dankwizard 1y ago

This is the part where you actually need to think and wonder if AI is the right tool in this particular purpose. Unfortunately you can't completely turn your brain off just yet.

wyager 1y ago

> god forbid you're researching a complex and possibly controversial subject and you want it to find reputable sources If you're really researching something complex/controversial, there may not be any