5 ms·

How to use generative AI for historical research: Four real-world case studies

37 points by benbreen 3y ago

jruohonen 3y ago
"I am starting to think that this sort of automated data visualization may end up being among the best use cases for AI in historical research." If you actually look at the plots, there are all kinds of horrors. "We just do it in a targeted way, because it’s time consuming." Please.
- mkii 3y ago
  I can see minor errors, and better opportunities for formatting/presentation. But "horrors"? A bit strong to call it that imo.
  - xwdv 3y ago
    Look closer.
  - drakenot 3y ago
    I looked at the first graph and it got a ton of the values wrong.
dartos 3y ago
The issue is always going to be verification and reproducibility with LLMs
- marginalia_nu 3y ago
  To be fair these are problems shared with historical research in general so in that way I guess it fits.
lysecret 3y ago
Very fascinating. I love hearing perspectives like this. This is more evidence for me that the medium term future will be extremely narrow agents that do one thing and do it well.
BartjeD 3y ago
Horrible popup on the blog, cant cancel it pn mobile
vunderba 3y ago
For somebody extolling the uses of generative AI, they missed several glaringly factually incorrect errors in the chart generation from the cigarette advertisement in the very first visualization. This is actually my biggest fear – that people are going to assume the infallibility of these agents and never fact check them.
- simplify 3y ago
  People already take random posters on social media as unchecked truth, yeah it's going to get much worse.
wolverine876 3y ago
> When asked to transcribe and translate those three lines at the bottom, the AI got the general idea, but also made significant errors > GPT-4 is quite useful for “getting the gist” of written sources in unfamiliar languages. People don't like to deal with tedious details and issues like accuracy (perhaps separating the good historians from the bad). They seem to wish - and seeing general agreement from fellow dreamers and dismissing tiresome objectors they seem to believe - that somehow misinformation won't be a problem. I don't know how, but somehow, because everyone does it, everyone believes it. As if the zeitgeist of the post-truth era will save them and us from the consequences. As if accuracy (completeness, correctness, consistency) of information is like cooking - for most dishes, overcooked or undercooked isn't great but it's fine. Information doesn't work that way: Unless you can read the original yourself - and if so, you don't need the GPT to translate - you have no idea if the GPT's 'gist' is correct - any element of it could be wrong, and likely some are. And a small mechanical error can cause devestating misunderstandings, such as a 'not' being attached to the wrong word (and remember different languages have very different rules about word order). If the GPT translates something as 'to our surprise, Jones, not Smith, was the Soviet spy', what does the original really say? Was it Smith? Jones? Maybe they were 'not' surprised? The true, rational answer is that you have no idea what it says (without some outside information). Information where any element can be inaccurate is useless, and worse, dangerous: believing a falsehood is worse than thinking, 'I don't know'.
- amelius 3y ago
  Dreamers seem to do do fairly well in life, however.
- raincole 3y ago
  If it works well, I think it proves history research is just fiction writing. But people won't admit it.
epups 3y ago
Honestly, the use cases presented here are probably cautionary tales about what not to do. Asking questions about the distant past, from which the LLM likely has very little data in its dataset and we have no "ground truth", is just begging for hallucinations, which you can see happening all over in this post.
vasco 3y ago
Not to define how LLMs can be used but for me the best case for History is different. Basically the idea is that since we're training LLMs on available data from a time period, historians from the future will be able to understand nuances about culture in a way better way if they boot up an LLM trained in 2023 than any other way of querying records. Things like "what did people in 2023 think about X", "what were some 2023 fads", etc. Some of this can be deduced nowadays from historical records but I think this has the potential of allowing way deeper understanding. Also better than asking that from the 2623 LLM which will be polluted by whatever is the current culture then. Unfortunately it doesn't help us, only the future historians.
singularity2001 3y ago
we urgently need a "by ai" tag for text and images etc. this is one place where maybe even regulation would make sense: software should require all generated content to add a (hidden) "made by ai" tag. any medium which does not allow such meta tags?
- flir 3y ago
  Obvious problem: most successful uses include a human in the process to a greater or lesser extent (eg collaborative text editing with a chatbot). What do you do with these in-between cases that actually make up the bulk of AI-generated works? I suggest a better approach is attestation. I, a human, vouch for this work.
gentleman11 3y ago
I tried this and… half of what chat got says is a lie. Using it to help your research is dangerous if you trust it at all. I’ve stopped using it completely by now (some of the lies are extremely subtle and hard to catch, some implied background detail it made up that totally skews your perception of a situation)

37 points by benbreen 3y ago

"I am starting to think that this sort of automated data visualization may end up being among the best use cases for AI in historical research." If you actually look at the plots, there are all kinds of horrors. "We just do it in a targeted way, because it’s time consuming." Please.

mkii 3y ago
I can see minor errors, and better opportunities for formatting/presentation. But "horrors"? A bit strong to call it that imo.
- xwdv 3y ago
  Look closer.
- drakenot 3y ago
  I looked at the first graph and it got a ton of the values wrong.

dartos 3y ago

The issue is always going to be verification and reproducibility with LLMs

marginalia_nu 3y ago
To be fair these are problems shared with historical research in general so in that way I guess it fits.

lysecret 3y ago

Very fascinating. I love hearing perspectives like this. This is more evidence for me that the medium term future will be extremely narrow agents that do one thing and do it well.

BartjeD 3y ago

Horrible popup on the blog, cant cancel it pn mobile

vunderba 3y ago

For somebody extolling the uses of generative AI, they missed several glaringly factually incorrect errors in the chart generation from the cigarette advertisement in the very first visualization. This is actually my biggest fear – that people are going to assume the infallibility of these agents and never fact check them.

simplify 3y ago
People already take random posters on social media as unchecked truth, yeah it's going to get much worse.

wolverine876 3y ago

> When asked to transcribe and translate those three lines at the bottom, the AI got the general idea, but also made significant errors > GPT-4 is quite useful for “getting the gist” of written sources in unfamiliar languages. People don't like to deal with tedious details and issues like accuracy (perhaps separating the good historians from the bad). They seem to wish - and seeing general agreement from fellow dreamers and dismissing tiresome objectors they seem to believe - that somehow misinformation won't be a problem. I don't know how, but somehow, because everyone does it, everyone believes it. As if the zeitgeist of the post-truth era will save them and us from the consequences. As if accuracy (completeness, correctness, consistency) of information is like cooking - for most dishes, overcooked or undercooked isn't great but it's fine. Information doesn't work that way: Unless you can read the original yourself - and if so, you don't need the GPT to translate - you have no idea if the GPT's 'gist' is correct - any element of it could be wrong, and likely some are. And a small mechanical error can cause devestating misunderstandings, such as a 'not' being attached to the wrong word (and remember different languages have very different rules about word order). If the GPT translates something as 'to our surprise, Jones, not Smith, was the Soviet spy', what does the original really say? Was it Smith? Jones? Maybe they were 'not' surprised? The true, rational answer is that you have no idea what it says (without some outside information). Information where any element can be inaccurate is useless, and worse, dangerous: believing a falsehood is worse than thinking, 'I don't know'.

amelius 3y ago
Dreamers seem to do do fairly well in life, however.
raincole 3y ago
If it works well, I think it proves history research is just fiction writing. But people won't admit it.

epups 3y ago

Honestly, the use cases presented here are probably cautionary tales about what not to do. Asking questions about the distant past, from which the LLM likely has very little data in its dataset and we have no "ground truth", is just begging for hallucinations, which you can see happening all over in this post.

vasco 3y ago

Not to define how LLMs can be used but for me the best case for History is different. Basically the idea is that since we're training LLMs on available data from a time period, historians from the future will be able to understand nuances about culture in a way better way if they boot up an LLM trained in 2023 than any other way of querying records. Things like "what did people in 2023 think about X", "what were some 2023 fads", etc. Some of this can be deduced nowadays from historical records but I think this has the potential of allowing way deeper understanding. Also better than asking that from the 2623 LLM which will be polluted by whatever is the current culture then. Unfortunately it doesn't help us, only the future historians.

singularity2001 3y ago

we urgently need a "by ai" tag for text and images etc. this is one place where maybe even regulation would make sense: software should require all generated content to add a (hidden) "made by ai" tag. any medium which does not allow such meta tags?

flir 3y ago
Obvious problem: most successful uses include a human in the process to a greater or lesser extent (eg collaborative text editing with a chatbot). What do you do with these in-between cases that actually make up the bulk of AI-generated works? I suggest a better approach is attestation. I, a human, vouch for this work.

gentleman11 3y ago

I tried this and… half of what chat got says is a lie. Using it to help your research is dangerous if you trust it at all. I’ve stopped using it completely by now (some of the lies are extremely subtle and hard to catch, some implied background detail it made up that totally skews your perception of a situation)