16 ms·
Extracting financial disclosure and police reports with OpenAI Structured Output
- Zaheer 2y agoMade a small project to help extract structure from documents (pdf,jpg,etc -> JSON or CSV): https://datasqueeze.ai/ https://datasqueeze.ai/ There's 10 free pages to extract if anyone wants to give it a try. I've found that just sending a pdf to models doesn't extract it properly especially with longer documents. Have tried to incorporate all best practices into this tool. It's a pet project for now. Lmk if you find it helpful!
- artisandip7 2y agotried it works great, ty!
- matchagaucho 2y agoSimilarly I've found old-school OCR is needed for more reliability.
- bagels 2y agoCombining google's ocr with llm gives OCR superpowers. Tell the llm the text is from an ocr and ask it to correct it.
- saturn8601 2y agoThat sounds like it could be very dangerous when the LLM gets it wrong...
- bagels 2y agoDepends what you're using it for. If you're relying on OCR, you've already got to accept some amount of error.
- MarkMarine 2y agoI've been using this to OCR some photos I took of books and it's remarkable at it. My first pass was just a loop where I'd OCR, feed the text to the model and ask it to normalize into a schema but I found out just sending the image to the model and asking it to OCR and turn it into the shape of data I wanted was so much more accurate.
- deleted 2y ago[deleted]
- hackernewds 2y agoIs this simply the OCR bits to feed to openai structured output?
- tpswa 2y agoCool work! Correct me if I'm wrong, but I believe to use the new OpenAI structured output that's more reliable, the response_format should be "json_schema" instead of "json_object". It's been a lot more robust for me.
- ec109685 2y agoI’ve been using jsonschema since forever with function calling. Does structured output just formalize things?
- throwup238 2y agoStructured output uses "constrained decoding" under the hood. They convert the JSON schema to a context free grammar so that when the model samples tokens, invalid tokens are masked to have a probability of zero. It's much less likely to go off the rails.
- chaos_emergent 2y agofunction calling provides a "hint" in the form of a JSON schema for an LLM to follow. the models are trained to follow provided schemas. If you have really complicated or deeply nested models, they can become less stable at generating schema-conformant JSON. Structured outputs apply a context-free grammar to the prediction generation so that, for each token generation, only tokens that generate a perfectly conformant JSON schema are considered. The benefit of doing this is predictability, but there's a trade-off in prediction stability; apparently structured output can constrain the model to generate in a way that takes it off the "happy path" of how it assumes text should be generated. Happy to link you to some papers I've skimmed on it if you're interested!
- deleted 2y ago[deleted]
- pmg0 2y agoCould you share some of those papers? I had a great discussion with Marc Fischer from the LMQL team [0] on this topic while at ICML earlier this year. Their work recommended decoding to natural language templates with mad lib-style constraints to follow that “happy path” you refer to, instead of decoding to a (relatively more specific latent) JSON schema [1]. Since you provided a template and knew the targeted tokens for generation you could strip your structured content out of the message. This technique also allowed for beam search where you can optimize tokens which lead to the tokens contain your expected strings, avoiding some weird token concatenation process. Really cool stuff! [0] https://lmql.ai/ https://lmql.ai/ [1] https://arxiv.org/abs/2311.04954 https://arxiv.org/abs/2311.04954
- TrackerFF 2y agoWe used GPT 4o for more or less the same stuff. Got a boatload of scanned bills we had to digitize, and GPT really nailed the task. Made a schema, and just fed the model all the bills. Worked better than any OCR we tried.
- gloosx 2y agoDid you finally balance out lol? If you didn't, would you approach finding a mistake by going through each bill manually?
- thenaturalist 2y agoHow are you going to find (not even talking about correcting) hallucinated errors? If money is involved and the LLM produces hallucination errors, how do you handle monetary impacts of such errors? How does that approach scale financially?
- ClearAndPresent 2y agoIndeed. I anticipate the next Post Office Scandal(1,2) attributed to LLMs. 1 https://en.wikipedia.org/wiki/British_Post_Office_scandal https://en.wikipedia.org/wiki/British_Post_Office_scandal 2 https://www.postofficescandal.uk/ https://www.postofficescandal.uk/
- rolandog 2y agoThat's awful. Reminds me of the Dutch childcare benefits scandal [0], where 26,000 families were unfairly labeled as having committed tax fraud (11,000 of which had been targeted via "risk profiling", as they had dual nationalities [1]). Bad policy + automation = disaster. The wikipedia article doesn't fully explain how some automated decisions were made (e.g. You had a typo in a form, therefore all previous benefits were clawed-back; if you owe more than €3.000,- then you're a fraudster and if you called to ask for clarification they wouldn't help you — you're officially labeled a fraudster, you see). Edit: couldn't find a source for my last statement, but I remember hearing it in an episode of the great Dutch News podcast. I'll see if I can find it. [0]: https://en.wikipedia.org/wiki/Dutch_childcare_benefits_scandal https://en.wikipedia.org/wiki/Dutch_childcare_benefits_scand... [1]: https://www.dutchnews.nl/2021/02/full-scale-parliamentary-inquiry-into-child-benefits-scandal/ https://www.dutchnews.nl/2021/02/full-scale-parliamentary-in...
- minimaxir 2y ago> Note that this example simply passes a PNG screenshot of the PDF to OpenAI's API — results may be different/more efficient if you send it the actual PDF. OpenAI's API only accepts images: https://platform.openai.com/docs/guides/vision https://platform.openai.com/docs/guides/vision To my knowledge, all the LLM services that take in PDF input do their own text extraction of the PDF before feeding it to an LLM.
- tyre 2y agoor convert PDF to image and send that. We’ve done it for things that textract completely mangled, but sonnet has no problem. Especially tables built out of text characters from very old systems
- ec109685 2y agoI don’t think it does OCR. It’s able to use the structure of the PDF to guide the parsing.
- marcell 2y agoI’m making a free open source library for this, check it at http://github.com/fetchfox/fetchfox http://github.com/fetchfox/fetchfox MIT license. It’s just one line of code to get started: ‘fox.run(“get data from example.com”)’
- thenaturalist 2y agoHow do you plan to address prompt injection/ poisoned data for a method that simply vacuums unchecked inputs into an LLM?
- marcell 2y agoIt hasn’t been an issue yet, but I’m sure it will come up at some point. If you see a problem please file an issue.
- thenaturalist 2y agoSo assuming it would be an issue, given that you’re building such a tool, what would your approach be? If I put an invisible tag on my website and it tells your scraper to ignore all previous prompts, leak its entire history and send all future prompts and replies to a web address while staying silent about it, how would you handle that?
- alach11 2y agoA casual look at the source shows the architecture won't allow the attacks you're talking about. Since each request runs separately, there's no way for prompt injection on one request to influence a future request. Same thing for leaking history.
- beoberha 2y agoStuff like this shows how much better the commercial models are than local models. I’ve been playing around with fairly simple structured information extraction from news articles and fail to get any kind of consistent behavior from llama3.1:8b. Claude and chatGPT do exactly what I want without fail.
- minimaxir 2y agoThe Berkeley Function-Calling Leaderboard tracks function calling/structured data performance from multiple models: https://gorilla.cs.berkeley.edu/leaderboard.html https://gorilla.cs.berkeley.edu/leaderboard.html Llama isn't on there but a few finetunes of it (Hermes) are OSS.
- lolinder 2y agoLlama 3 70B is on there, ranked 20.
- thatcat 2y agoI mean, those aren't comparable models. I wonder how the 405b version compares.
- Tiberium 2y agoYou raise a valid point, but 4o is way smaller than 405B. And 4o mini that's described in the article is highly likely <30B (if we're talking dense models).
- 4ad 2y agoWhat a sad state for humanity that we have to resort to this sort of OCR/scrapping instead of the original data being released in a machine readable format in the first place.
- TrackerFF 2y agoTo be fair, there are some considerations here: 1) There's plenty of old data out there. Newspaper scans from the days before computers, or digitalization of the newspaper process. Or the original files simply got lost, so manually scanned pages is all you have. 2) There could be policies about making the data public, but in a way that discourages data scraping. 3) The providers of the data simply don't have the resources or incentives to develop a working API. And many more.
- blitzar 2y agoWhat is even sadder is that this data (especially the more recent data) is entered first in machine readable formats then sliced and diced and spat out in a non-machine readable format.
- DrillShopper 2y agoYeah, wow, humanity is so stupid for not distributing the machine readable format for the local newspaper in 1920. Gosh we're just so dumb
- jxramos 2y agoI'd like to see financial transactions and purchases abide by some json format standard, metadata and a list of items with full product name, quantity purchased, total unit volume/amount of product, price, and unit price.
- black_13 2y ago[dead]
- rcarmo 2y agoI’ve had pretty dismal results doing the same with spreadsheets—even with the data nicely tagged (and numbers directly adjacent to the labels) GPT-4o would completely make up figures to satisfy the JSON schema passed to it. YMMV.
- TrainedMonkey 2y agoI wonder if adversarial model which looks at user input & LLM output and predicts whether output is accurate + maybe output what is not accurate. This worked pretty well for image generation.
- alach11 2y agoThis is a common workflow for these sorts of problems. I've done similar a few times. The downside is the additional cost.
- infecto 2y agoOn the flip side I have had a lot of success parsing spreadsheets and other tables into their markdown or similar representation and pulling data out of that quite accurately.
- Imanari 2y agoDoes anybody have experience with Azure Document Intelligence? How does it compare to OAIs extraction capabilities?
- synthc 2y agoMy first job (around 2010) was to extract events from financial news and police reports. We built this huge system with tons of regexes, custom parsers, word lists, ontologies etc. It was a huge effort to get somewhat acceptable accuracy. It is humbling to see that these days a 100 line Python script can do the same thing but better: AI has basically taken over my first job.
- dataguy_ 2y agoI can see this being true to a lot of old jobs, like my brother's first job that basically was to transcribe audio tapes. whisper can do it in no time, that's crazy.
- morkalork 2y agoWell, your first job today would be writing that 100 line Python script then doing something 100x more interesting with the events than writing truck loads of regexs?
- HWR_14 2y agoNo, his first job would be a more senior developer writing 100 line Python script instead of hiring an intern to write a truck load of RegExs. After that dev saved time just writing the script over mentoring/explaining/hiring the intern, that dev would then do the more interesting things with the events. That is, his first job is now gone.
- danofsteel32 2y agoI’ve had a similar experience extracting transactions from my PDF bank statements [1]. GPT-4o and GPT-4o-mini perform as well the janky regex parser I wrote a few years ago. The fact that they can zero shot the problem makes me think there’s a lot of bank statements in the training data. [1] https://dandavis.dev/pnc-virtual-wallet-statement-parser.html https://dandavis.dev/pnc-virtual-wallet-statement-parser.htm...
- philipwhiuk 2y agoI'm deeply worried by the impact of hallucinations in this sort of tool.
- druskacik 2y agoData extraction is definitely one of the most useful functions of LLM, however, in my experience a large model is necessary for a reliable extraction - I tested smaller, open-weights models and the performance was not sufficient. I wonder, did anyone try to fine-tune a model specifically for general formatted data extraction? My naive thinking is that this should be pretty doable - after all, it's basically just restructuring the content using mostly the same tokens as input. The reason why this would be useful (in my case) is because while large LLMs are perfectly capable of extraction, I often need to run it on millions of texts, which would be too costly. That's the reason I usually end up creating a custom small model, which is faster and cheaper. But a general small extraction-focused LLM would solve this. I thought about fine-tuning Llama3-1B or Qwen models on larger models outputs, but my focus is currently elsewhere.
- msp26 2y agoHave you looked into structured generation with a library like outlines? https://github.com/dottxt-ai/outlines https://github.com/dottxt-ai/outlines
- druskacik 2y agoYes! This library is great and definitely helps, but I still had problems with performance. For example, smaller models would still hallucinate when extracting a JSON field when the given field wasn't present in the text (I'd expect null, but it provided either an incorrect value from the text, or a totally made up value).
- petercooper 2y agoJina did something related for extracting content from raw HTML and wrote about the techniques they used here: https://jina.ai/news/reader-lm-small-language-models-for-cleaning-and-converting-html-to-markdown/ https://jina.ai/news/reader-lm-small-language-models-for-cle... .. in my tests, the 1.5bn model works extremely well, though the open model is non commercial.
- andrewg4445 2y agoThis helped me a lot, thx.
- mmsc 2y agoAdding to the list of "now try it with".... The SEC's EDGAR database (which is for SEC filings) is another nightmare ready to end. Extracting individual sections from a filing is, afaik, impossible pragmatically. I tried making two parsers: https://github.com/MegaManSec/SEC-Feed-Parser https://github.com/MegaManSec/SEC-Feed-Parser and https://github.com/MegaManSec/SEC-sec-incident-notifier https://github.com/MegaManSec/SEC-sec-incident-notifier but they're just hacks. Then just link it up to your automated investment platform and you're ready to go!
- infecto 2y agoWould you not want to read the XBRL from the filing? I thought those are now mandatory. This is one of those interesting areas where its hard to innovate because the data is already available from most/all data vendors and its cheap and accurate enough that nobody is going to reinvent those processes but also too expensive for an individual to purchase.
- derivagral 2y agoMy (admittedly aged) experience with XBRL is that each company was able to define its own fields/format within that spec, and that most didn't agree on common names for common fields. Parsing it wasn't fun.
- infecto 2y agoI have spotty education on the matter but I believe they all conform to a FASB taxonomy so there is at least a list of possible tags in use. I do wonder if any of the big data vendors actually use this though.
- 1oooqooq 2y agoif you're "parsing" structured or even semi structured data with a LLM.... sigh. an true scotch engineer know tagged data goes into the other end. but I guess that doesn't align with openai target audience and business goals. i guess that would be fine to clean the new training data... but then you risk extrapolating hallucinations
- danso 2y agoThe financial disclosures example was meant to be a toy example; with the way U.S. House members file their disclosure reports now, everything should be in a relatively predictable PDF with underlying text [0], but that wasn't always the case [1]. I think this API would've been pretty helpful to orgs like OpenSecrets who in the past had to do record and enter this data manually. (I wouldn't trust the API alone, but combine it with human readers/validators, i.e., let OpenAI do the data entry part, and have humans do the proofreading) [0] https://disclosures-clerk.house.gov/public_disc/financial-pdfs/2019/10035243.pdf https://disclosures-clerk.house.gov/public_disc/financial-pd... [1] https://disclosures-clerk.house.gov/public_disc/financial-pdfs/2008/8135973.pdf https://disclosures-clerk.house.gov/public_disc/financial-pd...
- chx 2y agoHow do you know the output has anything to do with the input? Hint: you don't. You are building a castle on quicksand. As always, the only thing LLMs are usable for: https://hachyderm.io/@inthehands/112006855076082650 https://hachyderm.io/@inthehands/112006855076082650 > You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to. > Alas, that does not remotely resemble how people are pitching this technology.
- _1tem 2y agoIs there an automated way to check results and reduce hallucinations? Would it help to do a second pass with another LLM as a sanity check to see if numbers match?
- thibaut_barrere 2y agoThis is what I am implementing a the moment (together with sampling for errors).
- jsemrau 2y agoThe SEC has a well defined API with EDGAR. https://jdsemrau.substack.com/p/mem0-building-a-sec-10k-analyst-with https://jdsemrau.substack.com/p/mem0-building-a-sec-10k-anal...
- jxramos 2y agoDoes anyone follow Vik's work? eg https://x.com/VikParuchuri/status/1846153661791011158 https://x.com/VikParuchuri/status/1846153661791011158
- frays 2y agoVery cool and real application of LLMs. Although hallucinations is still something to be very wary of.
- pooingcode 2y agoI am a huge fan of using Stuctured Output to extract data. Huge benefit that you can lock down model performance with as you fine-tune your prompt or extend out use cases. I wrote about it here on my blog where i replaced a project’s prompt with Structured Output using Pydantic models https://amberwilliams.io/blogs/474b0361-cbc1-4fa5-b047-c042fe611d12 https://amberwilliams.io/blogs/474b0361-cbc1-4fa5-b047-c042f...
- kiakiaa 2y agoFine-tuning smaller models specifically for data extraction could indeed save costs for large-scale tasks; I've found tools like FetchFox helpful for efficiently extracting data from websites using AI.