4 ms·

Looks cool, congrats! I've also worked with this data, but only for research purposes: https://www.finhist.com/bank-runs/episodes/13895.html https://www.finhi

Looks cool, congrats! I've also worked with this data, but only for research purposes: https://www.finhist.com/bank-runs/episodes/13895.html https://www.finhist.com/bank-runs/episodes/13895.html https://www.finhist.com/bank-runs/index.html https://www.finhist.com/bank-runs/index.html Surprisingly, I found out that layout was the trickiest thing, as newspaper articles often had multiple layers of headers, spanned multiple columns, etc. Do you have a preferred solution on that?

brettnbutter 5mo ago
Nice collection you have there. Just asked the Sleuth for some examples of that, and here's one to add to your Unional National one: https://www.finhist.com/bank-runs/episodes/19827.html https://www.finhist.com/bank-runs/episodes/19827.html https://snewpapers.com/components/0b22f0ca-60d2-4d63-be99-744aa4ab3142_evt_0?utm_source=x https://snewpapers.com/components/0b22f0ca-60d2-4d63-be99-74... Yes I agree the layouts are the trickiest part. I tried a few and ended up using some of the Paddle Paddle models for document layout analysis and orientation and such, which give bounding boxes and predicted reading order, but the reading orders aren't great even with SOTA most recent models on complex layouts, or even simple layouts when you have mastheads or images or other artifacts to work around. It's still valuable information that can be combined with heuristics though to stitch together a more accurate reading order, as the starting point of a pipeline
- zzleeper 5mo ago
  Great! Was thinking about PP but because I only ran an order of magnitude fewer articles (under 1mm pages; by piggybacking on Dell's OCR) I relied on Arcanum ( https://www.arcanum.com/en/newspaper-segmentation/about/ https://www.arcanum.com/en/newspaper-segmentation/about/ ) which was cheap enough (but I think not cheap enough at your scale). Cheers!
  - brettnbutter 5mo ago
    Hmm, I just tried to upload the jpgs of some of todays samples to Arcanum via https://www.arcanum.com/en/newspaper-segmentation/try-it/ https://www.arcanum.com/en/newspaper-segmentation/try-it/ and it didn't work. I'll try it again later, but it seems based on a cursory look that it wouldn't return info that I would need to correct it if I didn't like the output, and that I'd still have to stitch the individual pages back together myself? Probably much cheaper than my process though...

brettnbutter 5mo ago

Nice collection you have there. Just asked the Sleuth for some examples of that, and here's one to add to your Unional National one: https://www.finhist.com/bank-runs/episodes/19827.html https://www.finhist.com/bank-runs/episodes/19827.html https://snewpapers.com/components/0b22f0ca-60d2-4d63-be99-744aa4ab3142_evt_0?utm_source=x https://snewpapers.com/components/0b22f0ca-60d2-4d63-be99-74... Yes I agree the layouts are the trickiest part. I tried a few and ended up using some of the Paddle Paddle models for document layout analysis and orientation and such, which give bounding boxes and predicted reading order, but the reading orders aren't great even with SOTA most recent models on complex layouts, or even simple layouts when you have mastheads or images or other artifacts to work around. It's still valuable information that can be combined with heuristics though to stitch together a more accurate reading order, as the starting point of a pipeline

zzleeper 5mo ago
Great! Was thinking about PP but because I only ran an order of magnitude fewer articles (under 1mm pages; by piggybacking on Dell's OCR) I relied on Arcanum ( https://www.arcanum.com/en/newspaper-segmentation/about/ https://www.arcanum.com/en/newspaper-segmentation/about/ ) which was cheap enough (but I think not cheap enough at your scale). Cheers!
- brettnbutter 5mo ago
  Hmm, I just tried to upload the jpgs of some of todays samples to Arcanum via https://www.arcanum.com/en/newspaper-segmentation/try-it/ https://www.arcanum.com/en/newspaper-segmentation/try-it/ and it didn't work. I'll try it again later, but it seems based on a cursory look that it wouldn't return info that I would need to correct it if I didn't like the output, and that I'd still have to stitch the individual pages back together myself? Probably much cheaper than my process though...