8 ms·
Show HN: Open Paperless – Scan, index, and archive paper documents
- gravypod 9y agoWill this automatically center and apply perspective transforms to pictures taken with phone cameras?
- towndrunk 9y agoMicrosofts OfficeLen's for iOS is pretty nice doing this automagically.
- zhoubear 9y agoNot at the moment. I'm guessing this information is available in the EXIF properties?
- gravypod 9y agoNo. It requires opencv and image processing
- joshvm 9y agoIt's a little bit more tricky than that. What the EXIF might tell you is the camera calibration parameters like focal length, distortion, perspective center, etc. That can be used to fix systematic errors in images like pincushion/barrel distortion. To unwarp photos that were taken at odd angles you need to do some image processing. The mathematics aren't particularly difficult, it's a homography transform in most cases (rectangles). The problem is robustly detecting the page. Dropbox has some nice write-ups on this: https://blogs.dropbox.com/tech/2016/08/fast-document-rectification-and-enhancement/ https://blogs.dropbox.com/tech/2016/08/fast-document-rectifi...
- zhoubear 9y agoThanks for the link. That blew my mind! I wish it could be added would, my phone would replace my scanner instantly.
- gravypod 9y agoAlso see this for more implementation details: https://www.pyimagesearch.com/2014/09/01/build-kick-ass-mobile-document-scanner-just-5-minutes/ https://www.pyimagesearch.com/2014/09/01/build-kick-ass-mobi... This could be done server side if you're already doing ocr.
- zhoubear 9y agoWow this looks awesome, thank you!
- nothrabannosir 9y agoScannable (by Evernote ) does this wonderfully. Allows exporting as pdf and storing wherever. I’m a fan.
- grahamburger 9y agoGoogle Drive mobile app does this natively (at least on Android) there's a 'Scan' option when adding a file that will do it.
- ikurei 9y agoSince others are commenting alternative apps, I'll chime in: https://www.camscanner.com/ https://www.camscanner.com/ I've been using CamScanner on Android since forever. I only stopped a few months ago because I moved to iOS (and didn't realize it's also there). It does the page detection fairly well, but also let's you correct it. Very useful, specially when the picture is bad. I then used to upload it to Dropbox.
- voltagex_ 9y agoThe most complicated implementation I've seen has been Office Lens, which also corrects distortion and does edge detection.
- bvrlt 9y agoYou should check Genius Scan (for iOS and Android) that does that for you automatically, and much more: https://www.thegrizzlylabs.com/genius-scan/ https://www.thegrizzlylabs.com/genius-scan/ It also doesn't tie you to a specific ecosystem. Disclaimer: I'm on of the authors of Genius Scan :)
- gravypod 9y agoThere are many solutions for systems like this. Unfortunately none are open source. Most documents I do want to keep record of are sensitive. I don't trust a closed-source app with that kind of information.
- billbrown 9y agoThe best version of this that I've seen is Scanner Pro by Readdle. I had to scan three months worth of food receipts for an insurance claim and this feature was a lifesaver.
- bketelsen 9y agoseriously, thank you for mentioning this. Great app!
- ikawe 9y agoLet's put this in a room with [The Screenless Office](https://news.ycombinator.com/item?id=15960056 https://news.ycombinator.com/item?id=15960056) and see what happens.
- beamatronic 9y agoIs that the one where you scan a barcode and they mail you a printout of a web page ?
- carwyn 9y agoThere's also this paperless https://github.com/danielquinn/paperless https://github.com/danielquinn/paperless
- EGreg 9y agoI have a question Is there a service anyone knows about which will print your email and send it with tracking of receipt or signature, so you can prove what was physically sent? Or you mail it to them and they open your mail, scan it and forward it on with signature required, with your address as the return address? Because righy now you can only prove that the ENVELOPE was received, not what was in it.
- deleted 9y ago[deleted]
- craftyguy 9y agoyou can just ask your recipeint to send you some uuid on the letter after they receive it. i have no idea what problem you are trying to solve though.
- y4mi 9y agoi once heard of a case were the employer send an unrelated document to an employee. later on, the payment stopped and the employer claimed that they fired him at that time. I don't recall how that case ultimately turned out, but maybe something like that? would be incredibly rare though and of dubious worth for mostly anyone
- craftyguy 9y agoI believe there are courier services that help guarantee document delivery to the correct person, i.e. for sending/serving court summons.
- chrisweekly 9y agothere are lots of virtual mailbox service offerings out there, to open and scan mail for you. what you describe is right up their alley.
- prashnts 9y agoI've been using iOS and Mac's native notes app to do that. In my opinion what these solutions lack is an integration between both note-taking (I sometimes like to write a few sentences relevant to a document, and I'd like to have it shown right next to it) while also letting you have the individual documents available in PDF or whatever if you need. Notes app does it perfectly now after iOS 11.1 and High Sierra. An example is this screenshot from my notes https://imgur.com/a/xuZqW https://imgur.com/a/xuZqW
- zhoubear 9y agoRight now you can enter comments for a document that's already uploaded. A few UI tweaks could be added to show the latest comment under the document preview. Exporting to PDF sounds like a really good idea.
- lucaspiller 9y agoAs long as you don't mind being tied into the Apple ecosystem (and trust they won't loose your data) it's a good solution. Notes used to be backed by IMAP, but the new 'rich notes' since iOS 9/OS 10.11 is backed by iCloud.
- mark_l_watson 9y agoI used to use Notes but stopped after trying to back up my notes. For me, exporting one note at a time to pdf is not good enough, and finding the opaque binary file in ~/Library does not help because it is not a standard file format. I switched to using Notes in Fastmail.
- davidrupp 9y agoI've been using http://writeapp.net/notesexporter/ http://writeapp.net/notesexporter/ for Mac's Notes app; happy with it so far.
- X86BSD 9y agoI’ve been looking at “document management” software both open source and commercial for a very long time. Every one of them sucks. Are they powerful? Maybe. But their interfaces suck. Dependencies are out of control. Managing the app and it’s dependecnies are insanely time consuming. You want a product that scales, has a decent UI and the only way to get that is to make it simple. Also I require the “library” file to be stored on a network server not locally. The only app I have found to do these things is CDfinder. Everything else takes to much work to install, upgrade and maintain. Not to mention being too complex to reasonably manage, tag and search your photos, documents and digital data.
- SomewhatLikely 9y agoSomething I've wanted that might be possible is software that takes in a video of me flipping the pages of a notebook and converts that to a PDF of the notebook.
- mdaniel 9y agoI regret that I can't immediately find the video which discussed it, but this gets in the ballpark of what I saw: https://www.researchgate.net/publication/271462470_OCR_from_Video_Stream_of_Book_Flipping https://www.researchgate.net/publication/271462470_OCR_from_... IIRC, it wasn't ~vaporware~ researchware, but nor was it "clone this repo, away you go"
- curioussavage 9y agoAny good open source desktop software with linux support to do this? I don't see why I would personally want a web app for this.
- arca_vorago 9y agoThere are, I just can't think of them at the moment, I know though because I setup a bookscanner with a linux box. If I remember right the scan/ocr/archive tools are all seperate, so you would have to script them together.
- jackvalentine 9y agoI haven't tried this yet, but - https://openpaper.work/ https://openpaper.work/ Edit: tried it, it's crap.
- jk2323 9y agoMay I ask why? Installation is a bit cumbersome but it seems to be an outstanding program to me. I have been looking very long for something like this. I have not tired yet how it reacts to huge amounts of data. But best thing: NOT written in Java!
- jackvalentine 9y agoI installed it on Windows, so the installer was the best bit :) Maybe it's better on linux but it didn't use system dialogues, the UI behaved a bit strangely and it wasn't particularly intuitive. Maybe I'm just not the target - in a previous life I supported a HP TRIM ECM which may have left a mark on me.
- joelhaasnoot 9y agoIt's a little clunky but here's the one I found best that just worked on Ubuntu: http://gscan2pdf.sourceforge.net/ http://gscan2pdf.sourceforge.net/ . It can combine some of the best tools for OCR/cleanup/etc. My main gripe is that I have a document feeder and manually selecting pages with shift to combine in to a single document and clicking "Save as" is far too much of a hassle. There needs to be a better flow for that.
- rootsudo 9y agoOkay, wow, this is cool.
- mickael-kerjean 9y agoWell done! Will definitly give it a try back home !
- deleted 9y ago[deleted]
- pw0nka 9y agoLooks great. Love the idea behind it, but... There is at least one country (mine - Switzerland) which is not able to use software like yours. The problems are the current laws that force people and organizations to store physical copies of the documents (for several years). Electronic documents have no value in front of the law, which is why we have no choice but to do all of that offline, manually. I've tried many archiving solutions, but non of them saved any bit of time. The one single, missing feature was an automatism to print a serial code (the electronic document ID) back on the original document. This way you could just scan it, print it, put it in a large box where you sort it by its ID - that simple. And this would even work if you would use spacers to split the documents on the scanning process.
- anotheryou 9y agoThere is a proprietary solution for it, sadly overpriced and cloud-based. But the principle could be implemented in any software: Shoe Box + QR, you fill linearly and always snap the QR with the scan. It than can tell you at roughly which height in the stack and in which shoe box something is. edit: here: https://box.fileee.com/ https://box.fileee.com/
- pw0nka 9y agoThank you. This brings up another problem we have with our laws: Corporates are not allowed to store such documents on servers which are outside of our country. And that's usually the case with clouds services, because of obvious reasons.
- kuschku 9y agoThat’s not a problem with your laws, but more a problem with SaaS. Data should always stay as local as possible, ideally even within of your own organization.
- myaso 9y agoHow is keeping data local a good idea? Given how often there is a major breach reported, everything points to this being the opposite -- there aren't enough security specialists to go around for every company. Why not take advantage of the work done by the major cloud providers or SaaS companies instead of redoing everything in house all the time. It's like people want to actually solve the same problems over and over for eternity.
- y4mi 9y agoa nontrivial name conflict with Paperless (https://github.com/danielquinn/paperless https://github.com/danielquinn/paperless) ...
- andruby 9y agoThey even share the exact same tagline: "Scan, index, and archive all of your paper documents". If Open Paperless some kind of fork of Paperless?
- JustSomeNobody 9y agoThis had to be on purpose, which is quite disappointing.
- aminoson 9y agoIf you had been following the story of these projects you would know that it is paperless (and not open paperles or mayan) that has been leaching off other projects and people.
- y4mi 9y agohow did you get that idea? and why would DanielQuinn even bother? He doesnt accept any donations and instead encourages people to donate to charity... First commits: * open Paperless Dec 18, 2017 * Paperless Dec 20, 2015 * mayan edms Feb 3, 2011
- tjoff 9y agoI don't know what Mayan EDMS is and all this readme does is saying what it is in relation to Mayan EDMS. Extremely frustrating.
- Fnoord 9y agoAccording to the documentation it is Ubuntu only, as it requires Ubuntu 16.10 or later. What about other Linux distributions? No mention of the other 2 popular desktop OSes, Windows and macOS?
- pingec 9y agoDoes anyone know any similar free/open products for archiving documents, tracking etc.? What I am after is a system like expensive solutions have in some companies where the mailbox department prints (or has preprinted) labels with unique bar codes, for any incoming mail, they open it, stick a label on it, scan it with the label on it and then physically deliver it. Some departments also input recipient and sender details, add tags etc. So in the end they have a searchable database by persons involved, content type, tags and also all documents (physical and digital) have a referenceable id that can be used for various purposes.
- jopsen 9y agoQuestion: why bother organizing papers? I just throw everything in a box, if I ever need it again later it'll take a long time to find.. but I rarely need to find a document again. Complexity of archiving a document is O(1) with a very small constant. Complexity of retrieval is O(N) for a large N. But I have few retrievals in my system, so why pay a higher per document cost?
- pjc50 9y agoIf you have zero retrievals, you can just bin everything. As you've said, it depends how often you want to retrieve something and whether you ever need to retrieve a number of things in a hurry.
- phreeza 9y agoI do basically this but I store everything chronologically in big binders. Insertion is still O(1) and finding things is O(log n) if I know the date the document was generated. I keep large amounts of documents I never retrieve, and plan to do some culling in the future, but not having to agonize if something is worth keeping is worth it to me.
- pymai 9y ago- if you need to retrieve things often - if you need more than one box. will you still have a single box 10 years from now or will it be 2? good luck trying to retrieve something in 20 years when you have even more boxes - it only takes 1 fire or flood to destroy all of your documents. once they are digital you can easily make copies and store them in a few different locations tbh making digital copies isnt that complicated compared to throwing something in a box. i just scan everything into a year/month folder and do as you do... worry about finding them later. spending time tagging or naming stuff after theyve been digitised is optional.
- sschueller 9y agoI like this idea. I only need to keep 10 years and almost never need to retrieve anything. So if I use 10 boxes I can just throw away the oldest.
- bob_theslob646 9y agoPlease correct me if I am wrong, but this looks like you have to "name" each page. I would also want to see how accurate the ocr is. Historically, ocr on handwritting has been a problem unless the data is perfectly formatted. I guess the case is just to get enough accuracy so that you can look for or at the image of that page with the indexed search term you were looking for.
- mauritzio 9y agoMaybe it would be better to "archive" on good paper (encoded) Can not imagine a 1000 year old magnetic device... ;)
- Spearchucker 9y agoThis is arguably a lot more than I need. I'm a hoarder in that I have every email I've ever sent or received (bar junkmail), and every piece of paper I've ever received. Most of my paper is now scanned - I think I have two boxes left in my garden shed. I don't bother with OCR because search doesn't help me when I don't know what to search for (e.g. invoice for a jumper I bought in 2010 - fashion labels rarely call their jumpers jumper). And so I rely on meta data. There's not much out there in terms of open-source tagging software, and even less in terms of an open tagging approach. I ended up with tagspaces, which is a web app packaged up as a native app. The approach to tagging is good (tags appended to file name), but the app is abysmally poor. Slow - waiting up to 30 seconds for a pop-up menu to appear. It assumes tag-based searches work in only one way. The intent is to write some native apps to solve my biggest problems. For now I'm still trying to clear the backlog of un-scanned paper docs (not going to get this done for me, because privacy). I tag important stuff, like employment contracts, mortgage agreements, passports and birth certificates... Hope to have everything done by the time I cash in my chips. Might make for a useful dataset for someone somewhere some day.
- ishi 9y agoA few years ago I was involved with a startup that built a document management system for consumers, and we actually got pretty good results with OCR + automatic tagging based on a very simple database that maps keywords to tags. Let's say you want to auto-tag bills and other documents from your ISP. So you add the ISP's name, phone number, website address etc. into the database - any uniquely-identifying keywords that typically appear on the documents that they send. Now any document that contains these keywords will get tagged as "ISP", making it very easy to find in the future. Even if the OCR quality isn't perfect, at least one of these keywords will most likely get matched. Another example - you could add the names of your family members as keywords, making it easy to find all documents related to Jenny or Susan. You could argue that full-text search would achieve the same result, but uploading documents into the system and having them auto-tagged as "ISP", "car-payments", "Walmart", "Susan" and so on feels a little bit like magic, as if the system is actively helping you organize your papers. The keyword approach is also very easy to understand and tweak, unlike more rigorous but opaque methods of document clustering (such as tf-idf).
- theomega 9y agoI want to show an alternative approach to managing your documents: Store them in your IMAP/Mails. Either on an own account or in a dedicated sub-folder. I wrote some small python scripts [1] which allow you to: - Add an email with the PDF attached to your document collection. The script supports adding a subject and adding tags to it - Go over all the emails and run an OCR (tesseract) on them: Attach the OCR result together with the pdf to the email. Big advantage: - Search on IMAP is a solved problem - Clients for every operating system in the world, including web, mobile - Super simple backup and restore Over course, very geeky, nothing for your parents, but maybe something for you? [1]: https://github.com/theomega/IMAP_DMS https://github.com/theomega/IMAP_DMS
- JohnStrange 9y agoDon't you have to run your own IMAP server for that to work? Although my mail provider is fairly generous about storage space, it's not unlimited.
- theomega 9y agoDepends on two things: Your space and your privacy requirements. Google Mail works for example if you are willing to trust Google. A lot of email providers offer you a lot of space.
- mdaniel 9y agoJust ensure you pick a "good" encoding scheme, because base64-ing PDFs will balloon that storage space in a very big hurry
- karinato 9y agoFor those wondering about the relationship between Mayan EDMS, Paperless and Open Paperless here is a story line summary of the saga. Roberto Rosario (the creator of Mayan) is a very well known name in the Django, Python, document management, maker, hacking, open health and open source in the goverment circles. - https://speakerdeck.com/siloraptor https://speakerdeck.com/siloraptor - https://en.wikipedia.org/wiki/Roberto_Rosario https://en.wikipedia.org/wiki/Roberto_Rosario - https://www.pycon.it/conference/p/roberto-rosario https://www.pycon.it/conference/p/roberto-rosario - http://pyvideo.org/djangocon-us-2014/liberation-and-modernization-of-government-legacy.html http://pyvideo.org/djangocon-us-2014/liberation-and-moderniz... - https://cpucadviceletters.org/login/?next=/ https://cpucadviceletters.org/login/?next=/ - https://twit.tv/shows/floss-weekly/episodes/253 https://twit.tv/shows/floss-weekly/episodes/253 - https://en.wikipedia.org/wiki/Mayan_(software) https://en.wikipedia.org/wiki/Mayan_(software) - https://www.youtube.com/watch?v=rubzEAojf-k https://www.youtube.com/watch?v=rubzEAojf-k Mayan EDMS was initially released in February 3, 2011 (Wikipedia and git log). In June 2015, Roberto gave a workshop in DjangoCon named From zero to paperless with Mayan EDMS (https://archive.is/FDpYS https://archive.is/FDpYS). Daniel Quinn (the creator of Paperless) also attended and presented at the same DjangoCon event (https://vimeo.com/135907408 https://vimeo.com/135907408) and 6 months later after working on it for several months (Daniel's own words), he released Paperless on December 20, 2015 (https://github.com/danielquinn/paperless/commits/master?after=af4623e60563f5e4328e87ec8027f79804f8d08a+559 https://github.com/danielquinn/paperless/commits/master?afte...). By January 24, 2016, Paperless had "exploded in popularity" (https://twitter.com/danielagquinn/status/691242822431830016 https://twitter.com/danielagquinn/status/691242822431830016). Both projects used Python, Django, same Django 3rd party apps like DjangoSuit, same document consumer model, same OCR engine, REST API, among other things. On the surface it appeared that Paperless was a copy of Mayan EDMS concepts and implementations without giving credit or mention. Many additions were planned for Paperless that were features and implementations already in Mayan (https://www.reddit.com/r/selfhosted/comments/44mh88/scan_index_and_archive_all_of_your_paper_documents/ https://www.reddit.com/r/selfhosted/comments/44mh88/scan_ind...). A separate point of contention was that the name "Paperless" had been in use by other projects much earlier that Daniel's Paperless (https://github.com/search?utf8=%E2%9C%93&q=paperless&type= https://github.com/search?utf8=%E2%9C%93&q=paperless&type=). Since there is no trademark on the name or description, other projects appeared with the same name and description (https://github.com/lrnt/paperless https://github.com/lrnt/paperless). On March 15, 2016, Daniel presented Paperless at CodeNode (https://skillsmatter.com/skillscasts/7843-intro-to-paperless https://skillsmatter.com/skillscasts/7843-intro-to-paperless). It was Daniel's February 27, 2016 tweet suggesting to be paid to work on Paperless that sparked the animosity between the users of the two projects (https://twitter.com/danielagquinn/status/703629488932970500 https://twitter.com/danielagquinn/status/703629488932970500). Many heated debates ensued. Even then, the main critique of Paperless remained technical, but lack of maturity and implemenation was described by one Reddit users as: "I've looked into paperless and it currently lacks a lot of...nearly well everything. Maybe in a year or two" (https://www.reddit.com/r/linux/comments/6m9evn/want_to_go_paperless_looking_for_dms/dk1cjz0/ https://www.reddit.com/r/linux/comments/6m9evn/want_to_go_pa...) On April 9, 2016, Daniel added a reference to Mayan to the documentation of Paperless (https://github.com/danielquinn/paperless/commit/674d54ec38783b02350c1371bdf0f412dd765ef0#diff-88b99bb28683bd5b7e3a204826ead112 https://github.com/danielquinn/paperless/commit/674d54ec3878...). On April 17, 2016, Daniel posted on his old twitter account: "It looks like my idea for Paperless wasn't all that unique. This other project uses a lot of the same tools: http://www.mayan-edms.com" http://www.mayan-edms.com" (https://twitter.com/danielagquinn/status/721726208606646272 https://twitter.com/danielagquinn/status/721726208606646272). On April 14, 2017, Daniel Quinn posted in his blog a summary of his experiences at DjangoCon Europe 2017 where he mentions meeting Roberto in person. He describes Roberto as a "rival geek" in what appears to be jest and uses positive adjectives to describe Roberto in the rest of the post. (https://danielquinn.org/blog/djangocon-2017/ https://danielquinn.org/blog/djangocon-2017/) On April 16, 2017 Daniel posted a tweet mentioning the popularity Paperless (https://twitter.com/danielagquinn/status/853701257051205632 https://twitter.com/danielagquinn/status/853701257051205632). The last release of Paperless is made on Sep 9, 2017. On Oct 18, 2017 Daniel posted: "I changed my Twitter name! This isn't me any more, so if you're looking for me, you should keep head over to @danielagquinn." (https://twitter.com/searchingfortao/status/920778623715610624 https://twitter.com/searchingfortao/status/92077862371561062...). Only 7 commits have been made to Paperless since with the last commit happening on Novermber 5, 2017. On December 18, 2017 a user named "zhoubear" anounced on Reddit's selfhoted "Open Paperless: Scan, index, and archive all of your paper documents" (https://www.reddit.com/r/selfhosted/comments/7kjocg/scan_index_and_archive_all_of_your_paper_documents/ https://www.reddit.com/r/selfhosted/comments/7kjocg/scan_ind...). It turned out that Open Paperless was a forked Mayan EDMS with cosmetic changes but with copyrights changed and no attribution to Mayan EDMS. After a much heated debate, copyrights and attributions were restored and the project's description has been updated to show that it is a new front end for Mayan among other usability changes meant for home users. In 4 days, Open Paperless has surpassed Mayan EDMS in popularity on Github. No posts or comments from Roberto can be found in reference of Paperless or Open Paperless. https://twitter.com/search?q=paperless%20from%3Asearchingfortao&src=typd https://twitter.com/search?q=paperless%20from%3Asearchingfor...
- hemendrasingh 9y agogithub is the best website.