8 ms·
Donut: OCR-Free Document Understanding Transformer
- vosper 3y agoI want to build an application that scans restaurant and café menus (PDFs, photos, webpages) to identify which items are vegetarian or vegan. Would this work for that? If not, I would love to hear peoples ideas and suggestions.
- 7moritz7 3y agoThe non quantized models look relatively large, 800 MB. So you'd probably need to do inference on a server and somehow monetize that. Sounds difficult
- kolinko 3y agoWith vegan you can’t estimate it 100% from menu alone - because the sauce and other minor ingredients can be animal based. If you want to do it, using “plant based” is probably better than “vegan”, and it’s always good to make sure your users are aware that the mark can be wrong and they should double-check with the waiter. As for your question - I didn’t play with Donut, but ocr+gpt or multimodal gpt4 once released should handle this smoothly.
- ada1981 3y ago20+ years vegan here. You could combine ingredient search, looking for symbols the actually designate vegan as some places do, along with long/lat data to determine what the restaurant actually is and then check it with a database you maintain. So I could scan a menu, and then ask the owner or server about certain dishes, and then crowdsource an updated It would be great if there was a standard API for all restaurants that included all menu items, prices, ingredients, preparation and sourcing information. I could be maintained like a wiki I suppose, and restaurants could be incentivized by including their restaurants.
- vosper 3y agoYeah crowd-sourcing updates is the way I'd like to go. I'm hoping people will submit photos of menus, because lots of bakeries, cafés etc don't have much of an online presence or keep their menus up-to-date. Trying to solve the problem of scanning through menus for multiple restaurants to find something a vegan or vegetarian can eat, and instead just showing all the individual menu options in the area as a list.
- ada1981 3y agoOf course, anytime someone scans a menu you will have gps info available, so you can include it automatically
- Alifatisk 3y agoWe have a similar idea, but mine includes a few other categories!
- amelius 3y agoI bet in a year or so you don't need a specialized app for it, but you just ask your phone whatever you want to know about anything around you, including menus.
- Alifatisk 3y agoGoogle maps is pretty close to that, I can already find lots of places or restaurants based on what I want to eat. And now, thanks to crows sourcing, you can filter by a range of options. The only problem is that the data is closely walled by Google and you can only access it through their api. What I want to create is a tiny search engine that collects all menus (somehow collected from images) and let users find and filter what they like, and even get recommendations nearby!
- amelius 3y ago> The only problem is that the data is closely walled by Google and you can only access it through their api. Perhaps if iOS would allow an AI to access the screen pixels directly ...
- tuardoui 3y agoYou should look at LayoutLM models for a NER task. Then your pipeline should look like : - Identity the menu sub structure (title, item list ...) - Classify each item with 2 labels. The training process is not hard, but the data gathering / cleaning / labelling can be a little long.
- vosper 3y agoThanks! I haven't heard of LayoutLM but something that can understand structure from a few examples could be just what I need.
- tkanarsky 3y ago> Donut: DOcumeNt Understanding Transformer Author: phew! I'm glad there's an 'n' in there somewhere
- saretup 3y agoIs there an online tool/piece of code that can find words like this in a string?
- IanCal 3y agoI'm sure everyone is kinda tired of this answer, but gpt4. At least I have the share thing now so those who want to avoid it don't have to see a big pasted output. https://chat.openai.com/share/25124525-0bad-4c13-ae5a-ae4beac60360 https://chat.openai.com/share/25124525-0bad-4c13-ae5a-ae4bea... 3.5 doesn't really get it, 4 does. There are some that aren't great but the context is pretty small and it can be a decent launching point.
- TeMPOraL 3y agoI took your prompt and tried to get it to generate a bit less serious-sounding backronyms. GPT-4 is good at this. Me, not so much. https://cloud.typingmind.com/share/b49794e0-4b2b-4213-ae2e-8f63a39acacd https://cloud.typingmind.com/share/b49794e0-4b2b-4213-ae2e-8... I'll grant one thing: the paper must go well with cheese.
- JasonFruit 3y agoThat was a beautiful misunderstanding, right out of 1960s scifi.
- leononame 3y agoThis made me laugh out loud on the toilet. The cheese ones are great, I really like FETA
- IanCal 3y agoFantastic. I tried a few dafter ones based on this https://chat.openai.com/share/d9a31442-96e5-4a18-92d9-d217c7304803 https://chat.openai.com/share/d9a31442-96e5-4a18-92d9-d217c7...
- AmazingTurtle 3y agoI tested it out with a bunch of personal documents. Results were disappointing. Did not match up with the promised scores, not even slightly. I think the traditional approach to scanning and classifying without AI/ML is the way to go, for the next 5 years at very least.
- mhitza 3y agoDo you (or anyone else) know which would be a good open source OCR for PDFs and images?
- lelandfe 3y agoOCRmyPDF is the typical answer: https://github.com/ocrmypdf/OCRmyPDF https://github.com/ocrmypdf/OCRmyPDF It uses Tesseract under the hood. Results tend to just be OK in my experience.
- xrd 3y agoTesseract is amazing. It is simple and generally good results.
- version_five 3y agoI was just playing with tesseract last week (I'd used it years ago) and wasn't too happy. I had a pretty simple pdf that was in what you could think of as an old typewritten font, but easily legible, and I got all kinds of word fragments and nonsense characters in the output. I know that high quality ocr systems include a language model to coerce the read text into the most probable words. Is tesseract just supposed to be the first stage of such a system? I'll note that when I put the tesseract output into chatgpt and prompted it saying it was ocr'd text and asking to clean it up, it worked very well.
- deleted 3y ago[deleted]
- ryanjshaw 3y agoThis is really cool if it delivers. I tried building an app to scan till receipts. The image to text APIs out there really don't perform as well as you'd think. AWS Text Extract performed far better than GCP and Azure equivalents and traditional OCR solutions, but it still made some really annoying errors that I had to fix with heuristics.
- deleted 3y ago[deleted]
- werdnapk 3y agoWas Tesseract one of the APIs you tried?
- ryanjshaw 3y agoYup, it was solid but not as good as AWS out of the box. IIRC preprocessing the image did help, but I didn't have enough time to spend on fleshing that out for an MVP. (I gave up on the project when I realised that recently introduced protection of personal information laws in my country would have made this project too risky to continue work on. The intention was to automatically extract spending habits from receipts to improve personal finance management.)
- DannyBee 3y agoUnfortunately, trying this out, it seems to be nowhere near the claimed quality. Definitely not ready for prime time. Feels like someone trying to throw a stake in the ground rather than releasing a quality product, honestly.
- siddiqi123 3y ago[dead]
- i2cmaster 3y agoI've started using Microsoft's TROCR (another transformer OCR model) to read the cursive in my pocket journal (I have a habit of writing programs there first while I'm out and then typing them in manually, I just focus better that way.) It's surprisingly accurate although you have to write your own program to segment the image into lines. I think with some fine tuning I could have the machine read my notebook with minimal corrections.
- driscoll42 3y agoHave you looked into Craft or EAST for segmenting the image into lines? Those two work decently.
- armchairhacker 3y agoThese OCR tools are bringing us closer to msPaint as a viable IDE
- xavriley 3y agoThere’s a model for music transcription (audio to midi) called MT3 which takes an end-to-end transformer approach and claims SOTA on some datasets. However, from my own research and comparing with other models it seems that MT3 is very prone to overfitting and the real world results are not as impressive. A similar story seems to be playing out in the comments here
- onnodigcomplex 3y agoWhat would you say is a good model for audio to midi transcription?
- aosmith 3y agoSo is this why IA had an outage? Timing is perfect.
- dkatz23238 3y agoAs a developer who has been building IDP solutions I can assert that although this model is a lot larger (more weights) than a Graph Neural Network on OCR tokens, industry standard before transformers, it outperforms given enough data. Depending on how heterogenous the data is usually 200 documents can reach human levels of accuracy on documents, scoring by levenshtein ratio. Smaller graph models could get away with using less data. The problem that the "traditional" approach had is the the quality of the OCR was the bottleneck for overall model performance. It amazes me how this problem shifted from a node classification problem to a image to text problem. Training on CPU was possible with GCN but not with Donut.
- nestorD 3y agoI will have to investigate this, I am dreaming of a system that can take a pdf scan of a book as input and produce one or more properly formated (headings, italic, bold, underline, etc) markdown files. In my tests, LLMs have proved very good at cleaning a raw OCR but they need formating information to get me all the way.
- jcuenod 3y agoIt's not ready to take a book, but I'm building an app that takes scans of book chapters/journal articles (which I often receive from my college library) and turns them into well formatted PDFs (with OCR, consistent margins, rotation...) https://fixpdfs.com https://fixpdfs.com
- nestorD 3y agoI don't think it supports a markdown / formated text export but it looks looks fantastic as far as pdf cleanup goes (i currently rely on adobe scan for that when i am working from a paper copy), I will try it soon.
- jcuenod 3y agoNo, it doesn't support markdown and it doesn't do analysis of headers/page numbers. It's mainly aimed at making academic PDFs better for reading and annotating (especially on iPad-like devices). Hoping to start charging for it at some point, but I'm still trialing it... Some users have expected it to "unwarp" bad scans, it also doesn't do that unfortunately. But that's a much harder problem to solve...
- nestorD 3y agoI found perspective transformation to be good enough for 90% of cases. Going further would be a lot of effort better spent elsewhere. Does it deal fine with illustrations and pdf compression?
- qingcharles 3y ago
- dowakin 3y agoIf you want to train the Donut, check out this notebook on Kaggle. It trains Donut to read plots for a competition. The notebook contains full pipeline for finetuning. https://www.kaggle.com/code/nbroad/donut-train-benetech https://www.kaggle.com/code/nbroad/donut-train-benetech