5 ms·

The ML ink-detection models aren't spitting out Greek text. They're just predicting ink locations, which can be calibrated and cross-checked by manual inspectio

The ML ink-detection models aren't spitting out Greek text. They're just predicting ink locations, which can be calibrated and cross-checked by manual inspection with the original. (E.g., an earlier article showed how ink particles showed up as a shift in texture.) They operate on a lower level than letters and words, so if the ink does correspond to Greek letters that come out to recognizable Greek words forming sensible passages, it's good evidence that the output is correct. Presumably, it's possible for errors to slip through, but human labelers can similarly make errors.

mmooss 2y ago
> Presumably, it's possible for errors to slip through, but human labelers can similarly make errors. That's often an argument for AI systems: Is it better than humans (than avoiding car accidents, reading text, reading x-rays, etc.). But in science we need observable evidence.
- StableAlkyne 2y ago
  > But in science we need observable evidence. The observable evidence here is the raw x-ray data. Based on the article, what is being done here was using ML to process that X-ray data, to create another set of output that the humans could analyze. I understand jumping on the anti-AI bandwagon, but this ain't ChatGPT. Using ML to process complex instrument output has been used since at least the 70s. This is much closer to statistics based approaches than it is to some random startup trying to sell your manager on some snake oil AI-assisted whatever. I don't know your background, so I'm sorry if I'm assuming anything. Any chance you articulate what specifically you don't like about the method they used to process the data, other than "It's ML?"
  - mmooss 2y ago
    > I understand jumping on the anti-AI bandwagon Why be an asshole? What do you accomplish? I now have zero interest in what you have to say. I stopped there.
    - StableAlkyne 2y ago
      No animosity was meant there, friend. Completely above board, I get that a lot of people see "ML" and jump to conclusions given the hype around "AI" in the last couple of years.
      - mmooss 2y ago
        > No animosity was meant there Lol. That only supports my original assessment. Bye!
- LegionMammal978 2y ago
  We have plenty of observable evidence that the output of these detection models correponds to real ink residue, at least on one of the scrolls where the distinct texture is most obvious [0]. The models definitely aren't making things up out of pure noise. And if they did, then it almost certainly wouldn't come out as identifiable Greek letters, since these models are strictly looking at local patches of texture to identify the presence or absence of ink. Again, these aren't just black boxes spitting out long passages of Greek text, everything past the initial ink detection is still done by hand. [0] https://scrollprize.org/firstletters#caseys-crackle-pattern https://scrollprize.org/firstletters#caseys-crackle-pattern
  - mmooss 2y ago
    The question is, do the researchers use the ML/AI system to discover and identify likely text - which could be great - or do they use its output conclusively? In fairness, the paper may answer that question but I don't have time now ...
- coffeebeqn 2y ago
  Things like OCR or detection ML models don’t generally have the same drawbacks / hallucinations as LLMs for example. They’re not 100% accurate but they also won’t return made up results. Detecting a pattern is very different from any kind of “reasoning”
  - ben_w 2y ago
    Two things: 1. OCR absolutely does "hallucinate", and the word better suits that domain of OCR than the domain of LLMs as OCR will sometimes "see" text that doesn't actually exist. 2. This isn't OCR, it's more like contrast enhancement on a CT scan. Looking at a the project website, there's other work to figure out the shape of the paper surface because that's not obvious or simple either. The output of this model is what might be given to an OCR, but given how noisy the picture in the article is, I suspect not.
ctrlp 2y ago
Very clever
teleforce 2y ago
I think this is where most of the ML/AI practitioners need to focus on. Instead of obsessing over other accuracy technique for fully autonomous detection they should focus on human assisted detection to improve the quality/value of the solution. In this case recall/sensitivity is the utmost important accuracy metric and try to get it closer to 100% if possible. Hence even if the ML/AI got it wrong (due to false positive detection), at least the human experts can have a look at it and hopefully eliminate any false positive. It also considerably reduce the burden of the upstream manual based inspection/automation since it is machine automated. This can also mitigate and hopefully prevent false negative detection because false negative detection will not has the opportunity for further inspection/verification by the human experts or trained inspectors. Essentially the AI/ML is functioning as the filter shifting through the massive data very quickly (with minimum or zero false negative), and then the results are then verified by the much slower human experts.

mmooss 2y ago

> Presumably, it's possible for errors to slip through, but human labelers can similarly make errors. That's often an argument for AI systems: Is it better than humans (than avoiding car accidents, reading text, reading x-rays, etc.). But in science we need observable evidence.

StableAlkyne 2y ago
> But in science we need observable evidence. The observable evidence here is the raw x-ray data. Based on the article, what is being done here was using ML to process that X-ray data, to create another set of output that the humans could analyze. I understand jumping on the anti-AI bandwagon, but this ain't ChatGPT. Using ML to process complex instrument output has been used since at least the 70s. This is much closer to statistics based approaches than it is to some random startup trying to sell your manager on some snake oil AI-assisted whatever. I don't know your background, so I'm sorry if I'm assuming anything. Any chance you articulate what specifically you don't like about the method they used to process the data, other than "It's ML?"
- mmooss 2y ago
  > I understand jumping on the anti-AI bandwagon Why be an asshole? What do you accomplish? I now have zero interest in what you have to say. I stopped there.
  - StableAlkyne 2y ago
    No animosity was meant there, friend. Completely above board, I get that a lot of people see "ML" and jump to conclusions given the hype around "AI" in the last couple of years.
    - mmooss 2y ago
      > No animosity was meant there Lol. That only supports my original assessment. Bye!
LegionMammal978 2y ago
We have plenty of observable evidence that the output of these detection models correponds to real ink residue, at least on one of the scrolls where the distinct texture is most obvious [0]. The models definitely aren't making things up out of pure noise. And if they did, then it almost certainly wouldn't come out as identifiable Greek letters, since these models are strictly looking at local patches of texture to identify the presence or absence of ink. Again, these aren't just black boxes spitting out long passages of Greek text, everything past the initial ink detection is still done by hand. [0] https://scrollprize.org/firstletters#caseys-crackle-pattern https://scrollprize.org/firstletters#caseys-crackle-pattern
- mmooss 2y ago
  The question is, do the researchers use the ML/AI system to discover and identify likely text - which could be great - or do they use its output conclusively? In fairness, the paper may answer that question but I don't have time now ...
coffeebeqn 2y ago
Things like OCR or detection ML models don’t generally have the same drawbacks / hallucinations as LLMs for example. They’re not 100% accurate but they also won’t return made up results. Detecting a pattern is very different from any kind of “reasoning”
- ben_w 2y ago
  Two things: 1. OCR absolutely does "hallucinate", and the word better suits that domain of OCR than the domain of LLMs as OCR will sometimes "see" text that doesn't actually exist. 2. This isn't OCR, it's more like contrast enhancement on a CT scan. Looking at a the project website, there's other work to figure out the shape of the paper surface because that's not obvious or simple either. The output of this model is what might be given to an OCR, but given how noisy the picture in the article is, I suspect not.

ctrlp 2y ago

Very clever

teleforce 2y ago

I think this is where most of the ML/AI practitioners need to focus on. Instead of obsessing over other accuracy technique for fully autonomous detection they should focus on human assisted detection to improve the quality/value of the solution. In this case recall/sensitivity is the utmost important accuracy metric and try to get it closer to 100% if possible. Hence even if the ML/AI got it wrong (due to false positive detection), at least the human experts can have a look at it and hopefully eliminate any false positive. It also considerably reduce the burden of the upstream manual based inspection/automation since it is machine automated. This can also mitigate and hopefully prevent false negative detection because false negative detection will not has the opportunity for further inspection/verification by the human experts or trained inspectors. Essentially the AI/ML is functioning as the filter shifting through the massive data very quickly (with minimum or zero false negative), and then the results are then verified by the much slower human experts.