9 ms·

Financial Statement Analysis with Large Language Models

573 points by mellosouls 2y ago

gptisking 2y ago
great paper
soloist11 2y ago
[dead]
davidw 2y ago
If that's a thing, does it become a thing learning how to write a "poisoned" statement that misleads an LLM but is still factual?
- SirLJ 2y ago
  It is already been done as quants analyze company statements for the last decade at least, counting positive and negative words, etc...
- Drakim 2y ago
  Maybe, but it sounds hard if there are multiple LLMs out there that people might use to analyze such text. Tricking multiple LLMs with a certain poisonous combination of words and phrases sounds a lot like having a file that hashes to the same hash from different hashing techniques. Theoretically possible but actually practically impossible.
- infecto 2y ago
  As someone alluded to, the narrative that management drives has been examined and studied many times over. What is management saying, what are they not saying, what are they saying but not loudly, what did they say before that they no longer speak about. There are insights to glean but nothing that is giving you an unknown edge. Sentiment analysis and the like go back well into the late 80s, early 90s.
dpflan 2y ago
I recalling seeing a LinkedIn post by Greg Diamos at Lamini, they shared analysis of earnings calls. There are links on HuggingFace and GitHub, here they are: - https://huggingface.co/datasets/lamini/earnings-calls-qa https://huggingface.co/datasets/lamini/earnings-calls-qa - https://huggingface.co/datasets/lamini/earnings-raw https://huggingface.co/datasets/lamini/earnings-raw - https://github.com/lamini-ai/lamini-earnings-calls/tree/main https://github.com/lamini-ai/lamini-earnings-calls/tree/main
deadmutex 2y ago
It would've been interesting to compare models with larger context windows, e.g. Gemini with 1m+ tokens and Claude Opus. Otherwise, the title maybe should've been Financial Statement Analysis with GPT-4.
- infecto 2y ago
  The study only captured the financial statements. I am unsure what a larger context window would buy you.
  - foota 2y ago
    You could dump all of a companies financial statements together, or dump all of a companies competitors in with it, for one.
    - infecto 2y ago
      That was outside the context of what the paper was studying.
      - yaj54 2y ago
        maybe the authors should have used a larger context window in order to output their paper ;-) I jest - a well contained and specified context is important for a paper. But also, questions like "it would be interesting to also try this other thing" is how new papers get written.
antimatter15 2y ago
Figure 3 on p.40 of the paper seems to show that their LLM based model does not statistically significantly outperform a 3 layer neural network using 59 variables from 1989. This figure compares the prediction performance of GPT and quantitative models based on machine learning. Stepwise Logistic follows Ou and Penman (1989)’s structure with their 59 financial predictors. ANN is a three-layer artificial neural network model using the same set of variables as in Ou and Penman (1989). GPT (with CoT) provides the model with financial statement information and detailed chain-of-thought prompts. We report average accuracy (the percentage of correct predictions out of total predictions) for each method (left) and F1 score (right). We obtain bootstrapped standard errors by randomly sampling 1,000 observations 1,000 times and include 95% confidence intervals.
- infecto 2y ago
  Was going to point out the same. Glad to have the paper to read but I don't think the findings are significant.
  - foota 2y ago
    I agree this isn't earth shattering, but I think the benefit here is that it's a general solution instead of one trained on financial statements specifically.
    - _se 2y ago
      That is not a benefit. If you use a tool like this to try to compete with sophisticated actors (e.g. all major firms in the capital markets space) you will lose every time.
      - foota 2y ago
        We come up with all sorts of things that are initially a step backwards, but that lead to eventual improvement. The first cars were slower than horses. That's not to suggest that Renaissance is going to start using Chat GPT tomorrow, but maybe in a few years they'll be using fine tuned versions of LLMs in addition to whatever they're doing today. Even if it's not going to compete with the state of the art models for something, a single model capable of many things is still useful, and demonstrating domains where they are applicable (if not state of the art) is still beneficial.
primitivesuave 2y ago
The area where I see this making the most transformational change is by enabling average citizens to ask meaningful questions about the finances of their local government. In Cook County, Illinois, there are hundreds of local municipalities and elected authorities, all of which are producing monthly financial statements. There is not enough citizen oversight and rarely any media attention except in the most egregious cases (e.g. the recent drama in Dolton, IL, where the mayor is stealing millions in plain view of the citizens).
- tchalla 2y ago
  It’s your assumption that the lack of oversight is because of too much information. How will you validate that hypothesis before you invest in a solution?
- apwell23 2y ago
  > citizens to ask meaningful questions about the finances of their local government. is there a demand for this. I live in cook country. I really don't want to ask these questions. Not sure what I get out of asking these questions other than anger and frustration.
  - Kon-Peki 2y ago
    Municipal bond rating agencies should be a client of such data. And if not the rating agencies, people who invest in municipal bonds.
  - m463 2y ago
    if all the citizens can ask these questions, I think it will make a difference. and of course, the follow-up questions. Like who.
    - vsuperpower2020 2y ago
      Then anything you plan is doomed from the start. If companies start slipping cyanide into their food it would take at least 20 years for people to stop buying it. Getting everyone to simply do your thing while they're busy with their own life is a fool's errand.
fsndz 2y ago
[flagged]
- Robotenomics 2y ago
  Do you have examples without signing up?
  - fsndz 2y ago
    unfortunately no, but you can check the free blog https://www.lycee.ai/blog/getting-started-with-dspy https://www.lycee.ai/blog/getting-started-with-dspy
chollida1 2y ago
So the history of this type of research as I know it was that we - started to diff the executives statements from one quarter to another. Like engineering projects alot of this is pretty standard so the starting point is the last doc. Diffing allowed us to see what the executives added and thought was important and also showed what they removed. This worked well and for some things still does, this is what a warrant canary does, but stopped generating much alpha around 2010ish. - simple sentiment. We started to count positive and negative words to build a poor mans sentiment analysis that could be done very quickly upon doc release to trade upon. worked great up until around 2013ish before it started to be gamed and even bankruptcy notices gave positive sentiment scores by this metric. - sentiment models. Using proper models and not just positive and negative word counts we built sentiment models to read what the executives were saying. This worked well until about 2015/2016ish in my world view as by then executives carefully wrote out their remarks and had been coached to use only positive words. Worked until twitter killed the fire hose, and wasn't very reliable as reputable news accounts kept getting hacked. I remember i think AP new's account got hacked and reported a bombing at the white house that screwed up a few funds. You also had Anne Hathaway news pushing up Berkshire Hathaway's share price type issues in this time period. - there was a period here where we kept the same technology but used it everywhere from the twitter firehose to news articles to build a realtime sentiment model for companies and sectors. Not sure it generates much alpha due to garbage in, garbage out and data cleaning issues. - LLMs, with about GPT2 we could build models to do the sentiment analysis for us, but they had to be built out of foundational models and trained inhouse due to context limitations. Again this has been gamed by executives so alot of the research that I know of is now targeted at ingesting the Financials of companies and being able to ask questions quickly without math and programming. ie what are the top 5 firms in the consider discretionary space that are growing their earnings the fastest while not yet raising their dividends and whose share price hasn't kept up with their sectors average growth.
- nebula8804 2y ago
  I have no window into this world but I am curious if you know anything about the techniques that investors used to short or just analyze Tesla stock during the production hell of 2017-2020? It was an interesting window in ways that firms use to measure as much of the company as they can from the outside. In fact was there any other stock that was as heaving watched during that time? Looking back at that era it seemed investors were too focused on the numbers and fundamentals, even setting up live feeds of the factories to count the number of cars coming out and thats the same feeling I get from your post. It seems like dumb analysis ie. analysis without much context. We now know from the recent Isaacson biography what was happening on the other side. The shorts failed to measure the clever unorthodox ways that Musk and co would take to get the delivery numbers up. For example: The famous Tent. Musk used a loophole in CA laws to set up a giant tent in the parking lot and allowed him to boost the production by eliminating entire bottlenecks from the factory design. There is also just the religious like fervor with which the employees wanted to beat the shorts. I dont think this can be measured no? It helped to get them past the finish line.
nicce 2y ago
What will happen if everyone starts using heavy statistical methods or LLMs to predict stocks prices? And buys stock based on them? Will it absolutely make everything unpredictable? Edit: assuming that they initially provide good predictions
- infecto 2y ago
  This has already been a thing since the late 80s.
  - nicce 2y ago
    It hasn't been accurate enough to be meaningful, nor enough data.
    - infecto 2y ago
      There has been plenty of work behind the scenes over the decades that has been meaningful.
    - deleted 2y ago
      [deleted]
- surfingdino 2y ago
  HFT guys won't touch GPT. The stakes are too high. If LLMs could give those guys an edge they'd be all over this tech.
  - berkes 2y ago
    Or rather: If LLMs could give those guys an edge, there's no way they'd share their edge-giving LLMs with anyone, least of all their competition and the plebs.
  - deleted 2y ago
    [deleted]
  - tempusalaria 2y ago
01100011 2y ago
Top story on HN because we all secretly think we can be the next Jim Simons when in reality we're a few months away from posting loss porn to /r/WSB. If standardized LLM models are used to analyze statements, expect the statements to be massaged in ways that produce more favorable results from the LLM.
- tempodox 2y ago
  Yep, Goodhart's law is universal.
- bbor 2y ago
  I can’t wait until there’s warnings on stock market apps like cigarettes and lottery tickets. Well actually I guess there are no warnings on lotto tickets, probably for the exact same reason as why the government doesn’t protect people from being scammed by hedge funds with way more info than they have: the government needs that revenue.
  - mb5 2y ago
    CFD trading apps have this in the UK (and possibly EU, but I can’t see whether ESMA kept the rule): “78% of retail investor accounts lose money when trading CFDs with this provider.” https://www.fca.org.uk/news/press-releases/fca-confirms-permanent-restrictions-sale-cfds-and-cfd-options-retail-consumers https://www.fca.org.uk/news/press-releases/fca-confirms-perm... Of course, there are a lot more ways to lose money than CFDs.
doctoboggan 2y ago
If this were to become widely used, I can imagine executives writing financial statements, running them through an LLM, and tweaking it until they get the highest predicted future outcome. This would make the measure almost immediately useless.
- oceanplexian 2y ago
  This is already how it works. Have you listened to an earnings call? Especially companies like Tesla? They are a dog and pony show to sell investors on the stock.
  - doctoboggan 2y ago
    I am not saying executives aren't currently trying to game the system. I am saying currently the best they can do is estimate how thousands of analysts will respond. If LLM analysts become wide spread then they would be able to run simulations in a feedback loop until their report is optimized.
- perkolator 2y ago
  Anyone who has read many financial statements would understand this whole idea is already useless outside of financial statement education. LLMs could help a person learn to understand financial statements better, that is it. There are not all these hidden gems in financial statements though that are being currently missed that language models are going to unearth. Financial statements are already intentionally vague and often intentionally misleading. Corporate CEO/PR/Marketing are already the masters of writing many words while saying absolutely nothing.
  - rafaelero 2y ago
    Still, if you use GPT-4 it gives you 60% of accuracy in predicting if it's going to go up and down, which is considerably better than median human forecasters. Stop being so dismissive and start reading the numbers.
jellicle 2y ago
People are going to lose a LOT of money using this when the LLM says "buy" and old-school humans who read the same statement say "sell". But up until that day, it will probably be cheaper.
dangerwill 2y ago
I guess this makes sense. Because while there should be some noise from the text translation into the internal representation of the financial data once ingested into the model, the authors purposefully re-formatted all the reports to be formatted consistently. That then should allow the model to essentially do less of the LLM magic and more plain linear regression of the financial stats. And often past performance does have an impact on future performance, up to a point. I wonder what the results would have been with still-anonymized but non-fully standardized statements. Still though, impressive.
- nvy 2y ago
  You can scrape the filings from EDGAR, which presents the statements in standardized format.
  - repeekad 2y ago
    fintool.com already has an LLM on top of EDGAR
nextworddev 2y ago
To everyone thinking they can sell a LLM wrapper based on this - this is a very tough domain. You will soon run into data, distribution, and low demand. Funds that would actually use this are already using it.
- hadlock 2y ago
  Most every company I know has had a drag and drop PDF to LLM tool since Q1 2023. It's not a phenomenally difficult product to code up.
  - nextworddev 2y ago
    Not talking about a Chat-with-PDF solution. Talking about some large scale SEC filing analysis and visualization solution.
avidphantasm 2y ago
Great. Humans no longer need to cook the books and can claim plausible deniability. The only problem is the hallucination errors could go against you as well as for you.
- caseyy 2y ago
  They may claim, but there is no such plausible deniability. Not for lawyers using AI hallucinations, not for Tesla drivers crashing into things with FSD, not for tax fraud. People are ultimately held responsible and accountable for the way they use AI.
einpoklum 2y ago
Let's just skip ahead to just implementing the script of Idiocracy verbatim and be done with it.
motoboi 2y ago
Please add "and the financial result of this analysis will count toward your annual bonus" to the prompt.
jrochkind1 2y ago
how do you distinguish enterprise-level question-answering from other kinds of question-answering?
dankai 2y ago
if you want to see successful "machine learning based financial statement analysis", check out my paper & thesis. its from 2019 and ranks #1 for the term on google and gs because it is the first paper that applies a range of machine learning methods to all the quantitative data in them instead of just doing nlp on the text. happy to answer questions paper https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3520684 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3520684 thesis https://ora.ox.ac.uk/objects/uuid:a0aa6a5a-cfa4-40c0-a34c-08f779bb4bda https://ora.ox.ac.uk/objects/uuid:a0aa6a5a-cfa4-40c0-a34c-08...
caseyy 2y ago
> In this section, we aim to understand the sources of GPT’s predictive ability. Oh boy... I wonder how a neural net trained with unsupervised learning has a predictive ability. I wonder where that comes from... Unfortunately, the article doesn't seem to reach a conclusion. > We implement the CoT prompt as follows. We instruct the model to take on the role of a financial analyst whose task is to perform financial statement analysis. The model is then instructed to (i) identify notable changes in certain financial statement items, and (ii) compute key financial ratios without explicitly limiting the set of ratios that need to be computed. When calculating the ratios, we prompt the model to state the formulae first, and then perform simple computations. The model is also instructed to (iii) provide economic interpretations of the computed ratios. Who will tell them how an LLM works and that the neural net does not calculate anything? It only predicts the next token in a sentence of a calculation if it's been loss-minimized for that specific calculation. It looks like these authors are discovering large language models as if they are some alien animal. When they are mathematically describable and really not so mysterious prediction machines. At least the article is fairly benign. It's about the type of article that would pass as research in my MBA school as well... It doesn't reach any groundbreaking conclusions except to demonstrate that the guys have "probed" the model. Which I think is good. It's uninformed but not very misleading.
- famouswaffles 2y ago
  >Who will tell them how an LLM works and that the neural net does not calculate anything? You don't understand LLMs as well as you think you do. Yes, the neural network calculates things. >It only predicts the next token in a sentence of a calculation if it's been loss-minimized for that specific calculation. No that's not necessary at all. https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mec... https://cprimozic.net/blog/reverse-engineering-a-small-neural-network/ https://cprimozic.net/blog/reverse-engineering-a-small-neura...
  - caseyy 2y ago
    I have heard of generalization vs memorization, but the article you shared is very high quality. Thank you. I do not think that SOTA LLMs demonstrate grokking for most math problems. While I am a bit surprised to read how little training is necessary to achieve grokking in a toy setting (one specific math problem), the domain of all math problems is much larger. Also, the complexity of an applied mathematics problem is much higher than a simple mod problem. That seems to be what the author of the first article you quoted thinks as well. Our public models fail in that large domain a lot. For example, with tasks like counting elements in a set (words in a paragraph). Not to mention that they fail in complex applied mathematics tasks. If they have been loss-minimized for that specific calculation to the point that they exhibit this phase change, then that would be an exception. But in the financial statement analysis article, the author says explicitly that there isn't a limitation on the types of math problems they ask the model to perform. This is very, very irregular, and there are no guarantees that model has generalized them. In fact, it is much more likely that it hasn't, in my opinion. In any case, thank you again for the article. It's just such a massive contrast with the MBA article above.
Animats 2y ago
How do they get a LLM to do arithmetic?
- JonChesterfield 2y ago
  By not worrying whether it got the answer right. As long as some numbers go in and some numbers come out, vibes are good enough.
- mewpmewp2 2y ago
  Not sure about them, but usually you would let LLM execute code to calculate it.
uncomplete 2y ago
Headline should be 'LLM can read (and understand) PDFs'
diziet 2y ago
The fact that the paper does not mention the word "hallucinations" in the full body text makes me think that the authors aren't fully familiar with the state of LLMs as of 2024.
thedudeabides5 2y ago
bait
worik 2y ago
I am disturbed to see so much enthusiasm here for "trading" Markets matter, and some speculation is useful, but the purpose of markets is not speculation. Obviously. If you want to make some money get trained and get a good salary. Save your money in safe assets if you want to get supper rich be super creative, ensure going broke will only effect you (i.e. do not do this while supporting a family), and found a firm. You will likely fail, but there is a chance of super wealth and a bigger chance of a wild ride that will be good for you Trading from the perspective of greed runs the risk of total destruction. Putting you in jail, maybe. Bankruptcy if not too unlucky. Many people out of work because of your misallocation, and if you do not care about that I'm not interested in you The financial system is a zero sum game. (The economy in general is not) There is always someone cleverer and they likely do not care if they crush you. International finance is a snake pit Friends, look after friends. Maximise happiness. Be honest, be ethical, be safe Live long and prosper
- Terretta 2y ago
  > the purpose of markets is not speculation Arguably, the purpose of markets is part price discovery, part liquidity, and arguably mostly to support economic growth and stability by channeling funds from savers to those who can invest them productively. > If you want to make some money get trained and get a good salary. Save your money in safe assets Sure, as long as you're not into dynamism.
- latency-guy2 2y ago
  You assert a lot of personal opinions. On top of bad "characterizations" of people who are not you. I don't think this is a signature of a person being honest, or trustworthy.
- flourpower471 2y ago
  >>> If you want to make some money get trained and get a good salary. Save your money in safe assets. You mean like the good salary you get working for a trading firm? I'm not sure what this comment you made is meant to be, but it reads like a blend of somebody who's high and a tik tok wellness influencer. Trading is not a zero sum game in the sense you intend to suggest it is. It is 0 sum only if all participants have the same trading horizon. The pool grows in the same way because it is linked to the economy. The markets are a variety of players with different requirements. Most transactions occur between parties who have different horizons. Yes the hft makes money over 5s and the pension fund loses it. But the pension fund is looking at the return over the next year, so the small loss to the hft is just a cost of acquisition.
DonsDiscountGas 2y ago
> A key research design choice that we make is to not provide any textual information (e.g., Management Discussion and Analysis) that typically accompanies financial statements. While textual information is easy to integrate, our primary interest lies in understanding the LLMs’ ability to analyze and synthesize purely financial numbers. We use this setup to examine several research questions Seems like an odd test for a large language model. There are tabular models out there
bbor 2y ago
Oh damn, I thought this was about real finances, turns out it’s just part of that weird “property” thing they do in NYC. I hope someone’s working on feeding their Ledger files into a structured language model… More substantively, LLMs are for linguistic tasks. That’s why I’m super super bullish (heh) on llms for decoding EEG data, and incredibly bearish on their ability to accurately model a corporation’s asset flow. I just don’t see how the confounding variables / motivating forces would be at all linguistic. This is basically using LLMs for super-advanced arithmetic
pfdietz 2y ago
This would be an excellent target for poisoning.
gptisking 2y ago
this is a great paper
ryzvonusef 2y ago
Here is a view of an accountant trying to use AI with a financial statement: https://www.youtube.com/watch?v=VxxmzoZTRW4 https://www.youtube.com/watch?v=VxxmzoZTRW4 there are other videos on their youtube channel on the more analytical aspect of it, I just decided to share the latest one. It's been a hit and miss for now, depending on the model used (chatgpt/gemini/clause etc.) the results can vary somewhat.
blacktechnology 2y ago
Is there any product using this theroy?
rulalala 2y ago
This research and the comments mentioning the need of being fast as hell connect with the human capacity of using the information. Just a hypothetical point: at some point only trading engines driven by AI could make a timely use of this source.
bionhoward 2y ago
Just don’t train em on GME, gets tunnel vision on the meme price spikes and ignores / gets worse on other stocks. Pretty funny
anonu 2y ago
From a first principles approach: it does not really make sense to use an LLM to do fundamental analysis directly. Maybe you can use an LLM to write some python code to do fundamental analysis. But skipping that model building step and just applying a language model to numbers does not make intuitive sense to me. I am surprised at the results in the paper. The biggest red flag is that the researcher are not sure why there is predictive ability in LLMs. Maybe they didn't control for some lookahead bias.
gromneer 2y ago
I don't have a horse in the race. I don't even know what financial statement analysis is. But it worries me that a novel reliance on these models for traditionally skilled labor jobs will turn into a dependence. These models use the built up experience of human practitioners to achieve similar results. But if a dependence grows, then there will be no more skilled human practitioners to further develop improved skills and knowledge for these jobs. Calcifying the skills in time.
- namaria 2y ago
  That wouldn't be a problem if the models actually worked for that. We don't miss cobblers. But these models only seem to perform these jobs on the surface. Enough that companies will try them and waste resources. This will just hurt the bottom line optimizer shops and boost the professionals doing quality work on the long run.
akrymski 2y ago
But GPT4 has been trained up to recent events, so you can't do rolling predictions using just historical data with it. If your LLM knows the future obviously it can predict well, even if the company name is hidden.
EternalFury 2y ago
So, LLMs can’t do basic arithmetic very well at all but people use them to analyze financial statements?
fudiso 2y ago
great paper.
deleted 2y ago
[deleted]
fudiso 2y ago
[flagged]

573 points by mellosouls 2y ago

gptisking 2y ago

great paper

soloist11 2y ago

[dead]

davidw 2y ago

If that's a thing, does it become a thing learning how to write a "poisoned" statement that misleads an LLM but is still factual?

SirLJ 2y ago
It is already been done as quants analyze company statements for the last decade at least, counting positive and negative words, etc...
Drakim 2y ago
Maybe, but it sounds hard if there are multiple LLMs out there that people might use to analyze such text. Tricking multiple LLMs with a certain poisonous combination of words and phrases sounds a lot like having a file that hashes to the same hash from different hashing techniques. Theoretically possible but actually practically impossible.
infecto 2y ago
As someone alluded to, the narrative that management drives has been examined and studied many times over. What is management saying, what are they not saying, what are they saying but not loudly, what did they say before that they no longer speak about. There are insights to glean but nothing that is giving you an unknown edge. Sentiment analysis and the like go back well into the late 80s, early 90s.

dpflan 2y ago

I recalling seeing a LinkedIn post by Greg Diamos at Lamini, they shared analysis of earnings calls. There are links on HuggingFace and GitHub, here they are: - https://huggingface.co/datasets/lamini/earnings-calls-qa https://huggingface.co/datasets/lamini/earnings-calls-qa - https://huggingface.co/datasets/lamini/earnings-raw https://huggingface.co/datasets/lamini/earnings-raw - https://github.com/lamini-ai/lamini-earnings-calls/tree/main https://github.com/lamini-ai/lamini-earnings-calls/tree/main

deadmutex 2y ago

It would've been interesting to compare models with larger context windows, e.g. Gemini with 1m+ tokens and Claude Opus. Otherwise, the title maybe should've been Financial Statement Analysis with GPT-4.

infecto 2y ago
The study only captured the financial statements. I am unsure what a larger context window would buy you.
- foota 2y ago
  You could dump all of a companies financial statements together, or dump all of a companies competitors in with it, for one.
  - infecto 2y ago
    That was outside the context of what the paper was studying.
    - yaj54 2y ago
      maybe the authors should have used a larger context window in order to output their paper ;-) I jest - a well contained and specified context is important for a paper. But also, questions like "it would be interesting to also try this other thing" is how new papers get written.

antimatter15 2y ago

Figure 3 on p.40 of the paper seems to show that their LLM based model does not statistically significantly outperform a 3 layer neural network using 59 variables from 1989. This figure compares the prediction performance of GPT and quantitative models based on machine learning. Stepwise Logistic follows Ou and Penman (1989)’s structure with their 59 financial predictors. ANN is a three-layer artificial neural network model using the same set of variables as in Ou and Penman (1989). GPT (with CoT) provides the model with financial statement information and detailed chain-of-thought prompts. We report average accuracy (the percentage of correct predictions out of total predictions) for each method (left) and F1 score (right). We obtain bootstrapped standard errors by randomly sampling 1,000 observations 1,000 times and include 95% confidence intervals.

infecto 2y ago
Was going to point out the same. Glad to have the paper to read but I don't think the findings are significant.
- foota 2y ago
  I agree this isn't earth shattering, but I think the benefit here is that it's a general solution instead of one trained on financial statements specifically.
  - _se 2y ago
    That is not a benefit. If you use a tool like this to try to compete with sophisticated actors (e.g. all major firms in the capital markets space) you will lose every time.
    - foota 2y ago
      We come up with all sorts of things that are initially a step backwards, but that lead to eventual improvement. The first cars were slower than horses. That's not to suggest that Renaissance is going to start using Chat GPT tomorrow, but maybe in a few years they'll be using fine tuned versions of LLMs in addition to whatever they're doing today. Even if it's not going to compete with the state of the art models for something, a single model capable of many things is still useful, and demonstrating domains where they are applicable (if not state of the art) is still beneficial.

primitivesuave 2y ago

The area where I see this making the most transformational change is by enabling average citizens to ask meaningful questions about the finances of their local government. In Cook County, Illinois, there are hundreds of local municipalities and elected authorities, all of which are producing monthly financial statements. There is not enough citizen oversight and rarely any media attention except in the most egregious cases (e.g. the recent drama in Dolton, IL, where the mayor is stealing millions in plain view of the citizens).

tchalla 2y ago
It’s your assumption that the lack of oversight is because of too much information. How will you validate that hypothesis before you invest in a solution?
apwell23 2y ago
> citizens to ask meaningful questions about the finances of their local government. is there a demand for this. I live in cook country. I really don't want to ask these questions. Not sure what I get out of asking these questions other than anger and frustration.
- Kon-Peki 2y ago
  Municipal bond rating agencies should be a client of such data. And if not the rating agencies, people who invest in municipal bonds.
- m463 2y ago
  if all the citizens can ask these questions, I think it will make a difference. and of course, the follow-up questions. Like who.
  - vsuperpower2020 2y ago
    Then anything you plan is doomed from the start. If companies start slipping cyanide into their food it would take at least 20 years for people to stop buying it. Getting everyone to simply do your thing while they're busy with their own life is a fool's errand.

fsndz 2y ago

[flagged]

Robotenomics 2y ago
Do you have examples without signing up?
- fsndz 2y ago
  unfortunately no, but you can check the free blog https://www.lycee.ai/blog/getting-started-with-dspy https://www.lycee.ai/blog/getting-started-with-dspy

chollida1 2y ago

So the history of this type of research as I know it was that we - started to diff the executives statements from one quarter to another. Like engineering projects alot of this is pretty standard so the starting point is the last doc. Diffing allowed us to see what the executives added and thought was important and also showed what they removed. This worked well and for some things still does, this is what a warrant canary does, but stopped generating much alpha around 2010ish. - simple sentiment. We started to count positive and negative words to build a poor mans sentiment analysis that could be done very quickly upon doc release to trade upon. worked great up until around 2013ish before it started to be gamed and even bankruptcy notices gave positive sentiment scores by this metric. - sentiment models. Using proper models and not just positive and negative word counts we built sentiment models to read what the executives were saying. This worked well until about 2015/2016ish in my world view as by then executives carefully wrote out their remarks and had been coached to use only positive words. Worked until twitter killed the fire hose, and wasn't very reliable as reputable news accounts kept getting hacked. I remember i think AP new's account got hacked and reported a bombing at the white house that screwed up a few funds. You also had Anne Hathaway news pushing up Berkshire Hathaway's share price type issues in this time period. - there was a period here where we kept the same technology but used it everywhere from the twitter firehose to news articles to build a realtime sentiment model for companies and sectors. Not sure it generates much alpha due to garbage in, garbage out and data cleaning issues. - LLMs, with about GPT2 we could build models to do the sentiment analysis for us, but they had to be built out of foundational models and trained inhouse due to context limitations. Again this has been gamed by executives so alot of the research that I know of is now targeted at ingesting the Financials of companies and being able to ask questions quickly without math and programming. ie what are the top 5 firms in the consider discretionary space that are growing their earnings the fastest while not yet raising their dividends and whose share price hasn't kept up with their sectors average growth.

nebula8804 2y ago
I have no window into this world but I am curious if you know anything about the techniques that investors used to short or just analyze Tesla stock during the production hell of 2017-2020? It was an interesting window in ways that firms use to measure as much of the company as they can from the outside. In fact was there any other stock that was as heaving watched during that time? Looking back at that era it seemed investors were too focused on the numbers and fundamentals, even setting up live feeds of the factories to count the number of cars coming out and thats the same feeling I get from your post. It seems like dumb analysis ie. analysis without much context. We now know from the recent Isaacson biography what was happening on the other side. The shorts failed to measure the clever unorthodox ways that Musk and co would take to get the delivery numbers up. For example: The famous Tent. Musk used a loophole in CA laws to set up a giant tent in the parking lot and allowed him to boost the production by eliminating entire bottlenecks from the factory design. There is also just the religious like fervor with which the employees wanted to beat the shorts. I dont think this can be measured no? It helped to get them past the finish line.

nicce 2y ago

What will happen if everyone starts using heavy statistical methods or LLMs to predict stocks prices? And buys stock based on them? Will it absolutely make everything unpredictable? Edit: assuming that they initially provide good predictions

infecto 2y ago
This has already been a thing since the late 80s.
- nicce 2y ago
  It hasn't been accurate enough to be meaningful, nor enough data.
  - infecto 2y ago
    There has been plenty of work behind the scenes over the decades that has been meaningful.
  - deleted 2y ago
    [deleted]
surfingdino 2y ago
HFT guys won't touch GPT. The stakes are too high. If LLMs could give those guys an edge they'd be all over this tech.
- berkes 2y ago
  Or rather: If LLMs could give those guys an edge, there's no way they'd share their edge-giving LLMs with anyone, least of all their competition and the plebs.
- deleted 2y ago
  [deleted]
- tempusalaria 2y ago

01100011 2y ago

Top story on HN because we all secretly think we can be the next Jim Simons when in reality we're a few months away from posting loss porn to /r/WSB. If standardized LLM models are used to analyze statements, expect the statements to be massaged in ways that produce more favorable results from the LLM.

tempodox 2y ago
Yep, Goodhart's law is universal.
bbor 2y ago
I can’t wait until there’s warnings on stock market apps like cigarettes and lottery tickets. Well actually I guess there are no warnings on lotto tickets, probably for the exact same reason as why the government doesn’t protect people from being scammed by hedge funds with way more info than they have: the government needs that revenue.
- mb5 2y ago
  CFD trading apps have this in the UK (and possibly EU, but I can’t see whether ESMA kept the rule): “78% of retail investor accounts lose money when trading CFDs with this provider.” https://www.fca.org.uk/news/press-releases/fca-confirms-permanent-restrictions-sale-cfds-and-cfd-options-retail-consumers https://www.fca.org.uk/news/press-releases/fca-confirms-perm... Of course, there are a lot more ways to lose money than CFDs.

doctoboggan 2y ago

If this were to become widely used, I can imagine executives writing financial statements, running them through an LLM, and tweaking it until they get the highest predicted future outcome. This would make the measure almost immediately useless.

oceanplexian 2y ago
This is already how it works. Have you listened to an earnings call? Especially companies like Tesla? They are a dog and pony show to sell investors on the stock.
- doctoboggan 2y ago
  I am not saying executives aren't currently trying to game the system. I am saying currently the best they can do is estimate how thousands of analysts will respond. If LLM analysts become wide spread then they would be able to run simulations in a feedback loop until their report is optimized.
perkolator 2y ago
Anyone who has read many financial statements would understand this whole idea is already useless outside of financial statement education. LLMs could help a person learn to understand financial statements better, that is it. There are not all these hidden gems in financial statements though that are being currently missed that language models are going to unearth. Financial statements are already intentionally vague and often intentionally misleading. Corporate CEO/PR/Marketing are already the masters of writing many words while saying absolutely nothing.
- rafaelero 2y ago
  Still, if you use GPT-4 it gives you 60% of accuracy in predicting if it's going to go up and down, which is considerably better than median human forecasters. Stop being so dismissive and start reading the numbers.

jellicle 2y ago

People are going to lose a LOT of money using this when the LLM says "buy" and old-school humans who read the same statement say "sell". But up until that day, it will probably be cheaper.

dangerwill 2y ago

I guess this makes sense. Because while there should be some noise from the text translation into the internal representation of the financial data once ingested into the model, the authors purposefully re-formatted all the reports to be formatted consistently. That then should allow the model to essentially do less of the LLM magic and more plain linear regression of the financial stats. And often past performance does have an impact on future performance, up to a point. I wonder what the results would have been with still-anonymized but non-fully standardized statements. Still though, impressive.

nvy 2y ago
You can scrape the filings from EDGAR, which presents the statements in standardized format.
- repeekad 2y ago
  fintool.com already has an LLM on top of EDGAR

nextworddev 2y ago

To everyone thinking they can sell a LLM wrapper based on this - this is a very tough domain. You will soon run into data, distribution, and low demand. Funds that would actually use this are already using it.

hadlock 2y ago
Most every company I know has had a drag and drop PDF to LLM tool since Q1 2023. It's not a phenomenally difficult product to code up.
- nextworddev 2y ago
  Not talking about a Chat-with-PDF solution. Talking about some large scale SEC filing analysis and visualization solution.

avidphantasm 2y ago

Great. Humans no longer need to cook the books and can claim plausible deniability. The only problem is the hallucination errors could go against you as well as for you.

caseyy 2y ago
They may claim, but there is no such plausible deniability. Not for lawyers using AI hallucinations, not for Tesla drivers crashing into things with FSD, not for tax fraud. People are ultimately held responsible and accountable for the way they use AI.

einpoklum 2y ago

Let's just skip ahead to just implementing the script of Idiocracy verbatim and be done with it.

motoboi 2y ago

Please add "and the financial result of this analysis will count toward your annual bonus" to the prompt.

jrochkind1 2y ago

how do you distinguish enterprise-level question-answering from other kinds of question-answering?

dankai 2y ago

if you want to see successful "machine learning based financial statement analysis", check out my paper & thesis. its from 2019 and ranks #1 for the term on google and gs because it is the first paper that applies a range of machine learning methods to all the quantitative data in them instead of just doing nlp on the text. happy to answer questions paper https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3520684 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3520684 thesis https://ora.ox.ac.uk/objects/uuid:a0aa6a5a-cfa4-40c0-a34c-08f779bb4bda https://ora.ox.ac.uk/objects/uuid:a0aa6a5a-cfa4-40c0-a34c-08...

caseyy 2y ago

> In this section, we aim to understand the sources of GPT’s predictive ability. Oh boy... I wonder how a neural net trained with unsupervised learning has a predictive ability. I wonder where that comes from... Unfortunately, the article doesn't seem to reach a conclusion. > We implement the CoT prompt as follows. We instruct the model to take on the role of a financial analyst whose task is to perform financial statement analysis. The model is then instructed to (i) identify notable changes in certain financial statement items, and (ii) compute key financial ratios without explicitly limiting the set of ratios that need to be computed. When calculating the ratios, we prompt the model to state the formulae first, and then perform simple computations. The model is also instructed to (iii) provide economic interpretations of the computed ratios. Who will tell them how an LLM works and that the neural net does not calculate anything? It only predicts the next token in a sentence of a calculation if it's been loss-minimized for that specific calculation. It looks like these authors are discovering large language models as if they are some alien animal. When they are mathematically describable and really not so mysterious prediction machines. At least the article is fairly benign. It's about the type of article that would pass as research in my MBA school as well... It doesn't reach any groundbreaking conclusions except to demonstrate that the guys have "probed" the model. Which I think is good. It's uninformed but not very misleading.

famouswaffles 2y ago
>Who will tell them how an LLM works and that the neural net does not calculate anything? You don't understand LLMs as well as you think you do. Yes, the neural network calculates things. >It only predicts the next token in a sentence of a calculation if it's been loss-minimized for that specific calculation. No that's not necessary at all. https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mec... https://cprimozic.net/blog/reverse-engineering-a-small-neural-network/ https://cprimozic.net/blog/reverse-engineering-a-small-neura...
- caseyy 2y ago
  I have heard of generalization vs memorization, but the article you shared is very high quality. Thank you. I do not think that SOTA LLMs demonstrate grokking for most math problems. While I am a bit surprised to read how little training is necessary to achieve grokking in a toy setting (one specific math problem), the domain of all math problems is much larger. Also, the complexity of an applied mathematics problem is much higher than a simple mod problem. That seems to be what the author of the first article you quoted thinks as well. Our public models fail in that large domain a lot. For example, with tasks like counting elements in a set (words in a paragraph). Not to mention that they fail in complex applied mathematics tasks. If they have been loss-minimized for that specific calculation to the point that they exhibit this phase change, then that would be an exception. But in the financial statement analysis article, the author says explicitly that there isn't a limitation on the types of math problems they ask the model to perform. This is very, very irregular, and there are no guarantees that model has generalized them. In fact, it is much more likely that it hasn't, in my opinion. In any case, thank you again for the article. It's just such a massive contrast with the MBA article above.

Animats 2y ago

How do they get a LLM to do arithmetic?

JonChesterfield 2y ago
By not worrying whether it got the answer right. As long as some numbers go in and some numbers come out, vibes are good enough.
mewpmewp2 2y ago
Not sure about them, but usually you would let LLM execute code to calculate it.

uncomplete 2y ago

Headline should be 'LLM can read (and understand) PDFs'

diziet 2y ago

The fact that the paper does not mention the word "hallucinations" in the full body text makes me think that the authors aren't fully familiar with the state of LLMs as of 2024.

thedudeabides5 2y ago

bait

worik 2y ago

I am disturbed to see so much enthusiasm here for "trading" Markets matter, and some speculation is useful, but the purpose of markets is not speculation. Obviously. If you want to make some money get trained and get a good salary. Save your money in safe assets if you want to get supper rich be super creative, ensure going broke will only effect you (i.e. do not do this while supporting a family), and found a firm. You will likely fail, but there is a chance of super wealth and a bigger chance of a wild ride that will be good for you Trading from the perspective of greed runs the risk of total destruction. Putting you in jail, maybe. Bankruptcy if not too unlucky. Many people out of work because of your misallocation, and if you do not care about that I'm not interested in you The financial system is a zero sum game. (The economy in general is not) There is always someone cleverer and they likely do not care if they crush you. International finance is a snake pit Friends, look after friends. Maximise happiness. Be honest, be ethical, be safe Live long and prosper

Terretta 2y ago
> the purpose of markets is not speculation Arguably, the purpose of markets is part price discovery, part liquidity, and arguably mostly to support economic growth and stability by channeling funds from savers to those who can invest them productively. > If you want to make some money get trained and get a good salary. Save your money in safe assets Sure, as long as you're not into dynamism.
latency-guy2 2y ago
You assert a lot of personal opinions. On top of bad "characterizations" of people who are not you. I don't think this is a signature of a person being honest, or trustworthy.
flourpower471 2y ago
>>> If you want to make some money get trained and get a good salary. Save your money in safe assets. You mean like the good salary you get working for a trading firm? I'm not sure what this comment you made is meant to be, but it reads like a blend of somebody who's high and a tik tok wellness influencer. Trading is not a zero sum game in the sense you intend to suggest it is. It is 0 sum only if all participants have the same trading horizon. The pool grows in the same way because it is linked to the economy. The markets are a variety of players with different requirements. Most transactions occur between parties who have different horizons. Yes the hft makes money over 5s and the pension fund loses it. But the pension fund is looking at the return over the next year, so the small loss to the hft is just a cost of acquisition.

DonsDiscountGas 2y ago

> A key research design choice that we make is to not provide any textual information (e.g., Management Discussion and Analysis) that typically accompanies financial statements. While textual information is easy to integrate, our primary interest lies in understanding the LLMs’ ability to analyze and synthesize purely financial numbers. We use this setup to examine several research questions Seems like an odd test for a large language model. There are tabular models out there

bbor 2y ago

Oh damn, I thought this was about real finances, turns out it’s just part of that weird “property” thing they do in NYC. I hope someone’s working on feeding their Ledger files into a structured language model… More substantively, LLMs are for linguistic tasks. That’s why I’m super super bullish (heh) on llms for decoding EEG data, and incredibly bearish on their ability to accurately model a corporation’s asset flow. I just don’t see how the confounding variables / motivating forces would be at all linguistic. This is basically using LLMs for super-advanced arithmetic

pfdietz 2y ago

This would be an excellent target for poisoning.

gptisking 2y ago

this is a great paper

ryzvonusef 2y ago

Here is a view of an accountant trying to use AI with a financial statement: https://www.youtube.com/watch?v=VxxmzoZTRW4 https://www.youtube.com/watch?v=VxxmzoZTRW4 there are other videos on their youtube channel on the more analytical aspect of it, I just decided to share the latest one. It's been a hit and miss for now, depending on the model used (chatgpt/gemini/clause etc.) the results can vary somewhat.

blacktechnology 2y ago

Is there any product using this theroy?

rulalala 2y ago

This research and the comments mentioning the need of being fast as hell connect with the human capacity of using the information. Just a hypothetical point: at some point only trading engines driven by AI could make a timely use of this source.

bionhoward 2y ago

Just don’t train em on GME, gets tunnel vision on the meme price spikes and ignores / gets worse on other stocks. Pretty funny

anonu 2y ago

From a first principles approach: it does not really make sense to use an LLM to do fundamental analysis directly. Maybe you can use an LLM to write some python code to do fundamental analysis. But skipping that model building step and just applying a language model to numbers does not make intuitive sense to me. I am surprised at the results in the paper. The biggest red flag is that the researcher are not sure why there is predictive ability in LLMs. Maybe they didn't control for some lookahead bias.

gromneer 2y ago

I don't have a horse in the race. I don't even know what financial statement analysis is. But it worries me that a novel reliance on these models for traditionally skilled labor jobs will turn into a dependence. These models use the built up experience of human practitioners to achieve similar results. But if a dependence grows, then there will be no more skilled human practitioners to further develop improved skills and knowledge for these jobs. Calcifying the skills in time.

namaria 2y ago
That wouldn't be a problem if the models actually worked for that. We don't miss cobblers. But these models only seem to perform these jobs on the surface. Enough that companies will try them and waste resources. This will just hurt the bottom line optimizer shops and boost the professionals doing quality work on the long run.

akrymski 2y ago

But GPT4 has been trained up to recent events, so you can't do rolling predictions using just historical data with it. If your LLM knows the future obviously it can predict well, even if the company name is hidden.

EternalFury 2y ago

So, LLMs can’t do basic arithmetic very well at all but people use them to analyze financial statements?

fudiso 2y ago

great paper.

deleted 2y ago

[deleted]

fudiso 2y ago

[flagged]