3 ms·

>tree-based approach to organize and summarize text data, capturing both high-level and low-level details. https://twitter.com/parthsarthi03/status/17531992332

>tree-based approach to organize and summarize text data, capturing both high-level and low-level details. https://twitter.com/parthsarthi03/status/1753199233241674040 https://twitter.com/parthsarthi03/status/1753199233241674040 processes documents, organizing content and improving readability by handling sections, paragraphs, links, tables, lists, page continuations, and removing redundancies, watermarks, and applying OCR, with additional support for HTML and other formats through Apache Tika: https://github.com/nlmatics/nlm-ingestor https://github.com/nlmatics/nlm-ingestor

osigurdson 3y ago
I don't understand. Why build up text chunks from different, non-contiguous sections?
- infecto 3y ago
  If those non-contiguous sections share similar semantic/other meaning, it can make sense from a search perspective to group them?
  - bageler 3y ago
    it starts to look like a graph problem
- kjqgqkejbfefn 3y ago
  On the level of paper, not everything is laid out linearly. The main text is often laid out in column, the flow can be be offset with pictures with a caption, additional text can be placed in inserts, etc ... You need a human eye to figure that out and this is the task nlm-ingestor tackles. As for the content, semantic contiguity is not always guaranteed. A typical example of this are conversations, where people engage in narrative/argumentative competitions. Topics get nested as the conversation advances, along the lines of "Hey, this remind me of ...". Building up a stack that can be popped once subtopics have been exhausted: "To get back to the topic of ...". This is explored at length by Kebrat-Orecchioni in: https://www.cambridge.org/core/journals/language-in-society/article/abs/kerbrat-orecchionicatherine-les-interactions-verbales-vol-i-paris-armand-colin-1990-pp-318/16BCCCE4EEF5B0177A6AB7CF477C4A26 https://www.cambridge.org/core/journals/language-in-society/... And an explanation is offered by Dessalles in: https://telecom-paris.hal.science/hal-03814068/document https://telecom-paris.hal.science/hal-03814068/document

osigurdson 3y ago

I don't understand. Why build up text chunks from different, non-contiguous sections?

infecto 3y ago
If those non-contiguous sections share similar semantic/other meaning, it can make sense from a search perspective to group them?
- bageler 3y ago
  it starts to look like a graph problem
kjqgqkejbfefn 3y ago
On the level of paper, not everything is laid out linearly. The main text is often laid out in column, the flow can be be offset with pictures with a caption, additional text can be placed in inserts, etc ... You need a human eye to figure that out and this is the task nlm-ingestor tackles. As for the content, semantic contiguity is not always guaranteed. A typical example of this are conversations, where people engage in narrative/argumentative competitions. Topics get nested as the conversation advances, along the lines of "Hey, this remind me of ...". Building up a stack that can be popped once subtopics have been exhausted: "To get back to the topic of ...". This is explored at length by Kebrat-Orecchioni in: https://www.cambridge.org/core/journals/language-in-society/article/abs/kerbrat-orecchionicatherine-les-interactions-verbales-vol-i-paris-armand-colin-1990-pp-318/16BCCCE4EEF5B0177A6AB7CF477C4A26 https://www.cambridge.org/core/journals/language-in-society/... And an explanation is offered by Dessalles in: https://telecom-paris.hal.science/hal-03814068/document https://telecom-paris.hal.science/hal-03814068/document