4 ms·

That's a project I have too. A few things I struggled with, would be curious to know how you solved them? - backups. I plan on using it to go paperless and kee

That's a project I have too. A few things I struggled with, would be curious to know how you solved them? - backups. I plan on using it to go paperless and keep tax returns etc. I can't afford to lose that info, but I can't come up with a robust enough backup plan. How are you doing it? - concurrency. It runs on a 8 years old home server, and when I scan multiple pages documents, the ocr slows down to a crawl. Have you run into this?

rhizome 9y ago
It runs on a 8 years old home server, and when I scan multiple pages documents, the ocr slows down to a crawl. Have you run into this? OCR is going to be CPU bound, so the solution seems obvious.
- coaxial 9y ago
  Indeed, I wasn't too clear in my comment. It slows down to a crawl because rather than queueing OCR, it does it all at once. 100 documents means 100 tesseracts running in parallel.
  - rhizome 9y ago
    I would look into implementing a workflow.
Mister_Snuggles 9y ago
Backups are one of those things that I don't do well at the moment. Fortunately, Mayan doesn't have any special requirements - basically back up the database and back up its media directory[0]. My ideal solution is to have everything backed up "in the cloud" plus have a backup at home. So far I've got the home backup happening - every machine uses 'attic' to back up onto my NAS. I'm still figuring out what the "in the cloud" side looks like. One thing you can do is keep copies of important documents elsewhere - like Dropbox, a USB drive in a safe deposit box, etc. It all depends on what technologies you trust and how much you trust them. For OCR, I can't say I've had much trouble with it. Mayan breaks the PDFs into pages, then feeds each page to tesseract. All of that happens in a worker process and I haven't found that it has a significant impact on the use of Mayan. The OCR section for the document will be incomplete for a while though. [0] http://mayan.readthedocs.io/en/v2.1.4/topics/backups.html http://mayan.readthedocs.io/en/v2.1.4/topics/backups.html
- coaxial 9y ago
  I don't trust dropbox, at all. I'd like to store encrypted backups online for the Mayan db and every document. For the OCR, my issue is that if I import several documents Mayan will feed them all to tesseract in parallel. Then I have two available cores, but 40 tesseract workers and the system comes down because it runs out of RAM and CPU. How did you set up your OCR?
  - Mister_Snuggles 9y ago
    I just used the instructions here[0], with some minor changes (pathnames, db server, etc). I've never seen more than two tesseract processes running on my two-core VM, no matter how many documents I give it. Maybe something changed at some point to limit the active workers to the number of cores. [0] http://mayan.readthedocs.io/en/stable/topics/deploying.html http://mayan.readthedocs.io/en/stable/topics/deploying.html
    - siloraptor 9y ago
      Mayan EDMS issue to add rate limit support to the OCR backend. https://gitlab.com/mayan-edms/mayan-edms/issues/287 https://gitlab.com/mayan-edms/mayan-edms/issues/287
      - Mister_Snuggles 9y ago
        I wonder if there was a change in celery that limits the number of active workers to the number of cores. This is the behavior I see on my instance.
        coaxial 9y ago
        That could explain it, it would be a nice addition

rhizome 9y ago

It runs on a 8 years old home server, and when I scan multiple pages documents, the ocr slows down to a crawl. Have you run into this? OCR is going to be CPU bound, so the solution seems obvious.

coaxial 9y ago
Indeed, I wasn't too clear in my comment. It slows down to a crawl because rather than queueing OCR, it does it all at once. 100 documents means 100 tesseracts running in parallel.
- rhizome 9y ago
  I would look into implementing a workflow.

Mister_Snuggles 9y ago

Backups are one of those things that I don't do well at the moment. Fortunately, Mayan doesn't have any special requirements - basically back up the database and back up its media directory[0]. My ideal solution is to have everything backed up "in the cloud" plus have a backup at home. So far I've got the home backup happening - every machine uses 'attic' to back up onto my NAS. I'm still figuring out what the "in the cloud" side looks like. One thing you can do is keep copies of important documents elsewhere - like Dropbox, a USB drive in a safe deposit box, etc. It all depends on what technologies you trust and how much you trust them. For OCR, I can't say I've had much trouble with it. Mayan breaks the PDFs into pages, then feeds each page to tesseract. All of that happens in a worker process and I haven't found that it has a significant impact on the use of Mayan. The OCR section for the document will be incomplete for a while though. [0] http://mayan.readthedocs.io/en/v2.1.4/topics/backups.html http://mayan.readthedocs.io/en/v2.1.4/topics/backups.html

coaxial 9y ago
I don't trust dropbox, at all. I'd like to store encrypted backups online for the Mayan db and every document. For the OCR, my issue is that if I import several documents Mayan will feed them all to tesseract in parallel. Then I have two available cores, but 40 tesseract workers and the system comes down because it runs out of RAM and CPU. How did you set up your OCR?
- Mister_Snuggles 9y ago
  I just used the instructions here[0], with some minor changes (pathnames, db server, etc). I've never seen more than two tesseract processes running on my two-core VM, no matter how many documents I give it. Maybe something changed at some point to limit the active workers to the number of cores. [0] http://mayan.readthedocs.io/en/stable/topics/deploying.html http://mayan.readthedocs.io/en/stable/topics/deploying.html
  - siloraptor 9y ago
    Mayan EDMS issue to add rate limit support to the OCR backend. https://gitlab.com/mayan-edms/mayan-edms/issues/287 https://gitlab.com/mayan-edms/mayan-edms/issues/287
    - Mister_Snuggles 9y ago
      I wonder if there was a change in celery that limits the number of active workers to the number of cores. This is the behavior I see on my instance.
      - coaxial 9y ago
        That could explain it, it would be a nice addition