4 ms·

I have not found any mention of accuracy. Since it's using LLM, how accurate the conversion is? As in does that NASA document match 100% with the pdf or did it

I have not found any mention of accuracy. Since it's using LLM, how accurate the conversion is? As in does that NASA document match 100% with the pdf or did it introduce any made up things (hallucinations)? That converted NASA doc should be included in repo and linked in readme if you haven't already.

yigitkonur35 2y ago
People are really freaked out about hallucinations, but you can totally tackle that with solid prompts. The one in the repo right now is doing a pretty good job. Keep in mind though, this project is all about maxing out context for LLMs in products that need PDF input. We're not talking about some hardcore archiving system for the Library of Congress here. The goal is to boost consistency whenever you're feeding PDF context into an LLM-powered tool. Appreciate the feedback, I'll be sure to add that in.
- deleted 2y ago
  [deleted]
- williamdclt 2y ago
  I don’t think any prompting skill guarantees the absence of hallucination. And if hallucination is possible, you will usually need to worry about it
- Foobar8568 2y ago
  As soon as you have something else than a paragraph in a single column layout, you will get hallucinations, random stuff, cut off etc even if you say which pages to look at, LLM will just do what ever.
- freedomben 2y ago
  Can you give some examples of prompts that you use that will tackle hallucinations?
- AdieuToLogic 2y ago
  > People are really freaked out about hallucinations, but you can totally tackle that with solid prompts. > The goal is to boost consistency whenever you're feeding PDF context into an LLM-powered tool. These two assertions are contradictory. There are no "solid prompts" which obviate anthropomorphic "LLM hallucinations." Also, there is no deterministic consistency when "feeding PDF context" into an intrinsically non-deterministic algorithm, as any "LLM-powered tool" is by definition.
- smusamashah 2y ago
  > hallucinations, but you can totally tackle that with solid prompts. This is so wrong. This so much sound as if you have not used LLMs to do any real work.
uLogMicheal 2y ago
Couldn't one just program a linear word matching test to ensure correctness?

yigitkonur35 2y ago

People are really freaked out about hallucinations, but you can totally tackle that with solid prompts. The one in the repo right now is doing a pretty good job. Keep in mind though, this project is all about maxing out context for LLMs in products that need PDF input. We're not talking about some hardcore archiving system for the Library of Congress here. The goal is to boost consistency whenever you're feeding PDF context into an LLM-powered tool. Appreciate the feedback, I'll be sure to add that in.

deleted 2y ago
[deleted]
williamdclt 2y ago
I don’t think any prompting skill guarantees the absence of hallucination. And if hallucination is possible, you will usually need to worry about it
Foobar8568 2y ago
As soon as you have something else than a paragraph in a single column layout, you will get hallucinations, random stuff, cut off etc even if you say which pages to look at, LLM will just do what ever.
freedomben 2y ago
Can you give some examples of prompts that you use that will tackle hallucinations?
AdieuToLogic 2y ago
> People are really freaked out about hallucinations, but you can totally tackle that with solid prompts. > The goal is to boost consistency whenever you're feeding PDF context into an LLM-powered tool. These two assertions are contradictory. There are no "solid prompts" which obviate anthropomorphic "LLM hallucinations." Also, there is no deterministic consistency when "feeding PDF context" into an intrinsically non-deterministic algorithm, as any "LLM-powered tool" is by definition.
smusamashah 2y ago
> hallucinations, but you can totally tackle that with solid prompts. This is so wrong. This so much sound as if you have not used LLMs to do any real work.

uLogMicheal 2y ago

Couldn't one just program a linear word matching test to ensure correctness?