4 ms·
I love it. Instead of (a) running mathematical experiments that model human beings as utility-maximizing agents in a highly-simplified toy economy (easy and che
by cs702 4y ago
I love it. Instead of (a) running mathematical experiments that model human beings as utility-maximizing agents in a highly-simplified toy economy (easy and cheap, but unrealistic), or (b) running large-scale social experiments on actual human beings (more realistic, but hard and expensive), the authors propose (c) running large-scale experiments on large language models (LLMs) trained to respond, i.e., behave, like human beings. Recent LLMs seem to model human beings well enough for it!
Abstract:
> Newly-developed large language models (LLM)—because of how they are trained and designed—are implicit computational models of humans—a homo silicus. These models can be used the same way economists use homo economicus: they can be given endowments, information, preferences, and so on and then their behavior can be explored in scenarios via simulation. I demonstrate this approach using OpenAI’s GPT3 with experiments derived from Charness and Rabin (2002), Kahneman, Knetsch and Thaler (1986) and Samuelson and Zeckhauser (1988). The findings are qualitatively similar to
the original results, but it is also trivially easy to try variations that offer fresh insights. Departing from the traditional laboratory paradigm, I also create a hiring scenario where an employer faces applicants that differ in experience and wage ask and then analyze how a minimum wage affects realized wages and the extent of labor-labor substitution.
- bmc7505 4y agohttps://link.springer.com/article/10.1007/s11229-020-02950-3 https://link.springer.com/article/10.1007/s11229-020-02950-3
- jmeister 4y agoGreat reference. Thank you. I think simulation is underused as a thinking aid and pedagogical tool. In statistics Andrew Gelman has been championing simulation lately.
- reilly3000 4y agoI believe in simulation for those reasons. I really enjoy the physics sims but be found online and they have helped me develop a better intuition on topics than my college courses did. That said, I don’t know how to apply it to my everyday life and work. I don’t know what kinds of inputs are required for meaningful models to be possible. I don’t even know where to start with tooling, other than raw scripting. Any suggestions for starting points?
- eternalban 4y ago> Recent LLMs seem to model human beings well enough for it! Human beings are bundles of emotions and feelings. Quite a lot of economic activity of humans is motivated by irrational impulses that are engendered in society and through interacting in society with other humans. More fundamentally, OP’s assertions regarding homo silicus and “implicit computational models of humans” are precisely the matter under contention. Does language fully capture human existence? Is thought truly simply a side effect of language? I am in the camp that says, no.
- cs702 4y agoNo one's saying that recent LLMs "fully capture human existence," whatever that may mean. But the evidence in this paper suggests they simulate human beings well enough for these kinds of experiments.
- eternalban 4y agoI’ll retract that “fully”. https://twitter.com/gdb/status/1611429677218004992 https://twitter.com/gdb/status/1611429677218004992
- cs702 4y ago:-)
- rcme 4y agoAre LLMs rational?
- eternalban 4y agotouche
- astrange 4y agoThat's not what "rational"/"irrational" means in economics. It means something like, if you're an economist and you see the price of eggs has spiked, you can't just enter the market at the old price and win all the business from existing egg farmers. You don't know more than existing market actors (they're "rational agents") just because you know economics.
- andrepd 4y agoAttempting to draw any kind of conclusions about the real world and human behaviour from a chatbot. Can't decide if this is hilarious or disturbing.
- john_horton 4y agoIt is so implausible that the training process that creates LLMs might learn features of human behavior that could then be uncovered via experimentation? I showed, empirically, that one can replicate several findings in behavioral economics with AI agents. Perhaps the model "knows" how to behave from these papers, but I think the more plausible interpretation is that it learned about human preferences (against price gouging, status quo bias, & so on) from its training. As such, it seems quite likely that there are other latent behaviors captured by LLMs and yet to be discovered.
- jcampbell1 4y agoThe potential hole in your thinking is the end of your paper where you advise how to get good answers: ask questions in an economist phd style! This presents a problem left unaddressed.
- john_horton 4y agoAre you referring to this: "What kinds of experiments are likely to work well? Given current capabilities, games with complex instructions are not presently likely to work well, but with more advanced LLMs on the horizon, this is likely to change. I should also note that research questions like what is “the effect of x on y” are likely to work much better than questions like “what is the level of x?.” Consider that in my Kahneman et al. (1986) example, I can create AI “socialists” who are not too keen on the price system generally. If I polled them about who they want for president, there is no reason to think it would generalize to the population at large. But if my research question was “what is the effect of the size of the price increase on moral judgments” I might get be able to make progress. That being said, it might be possible to create agents with the correct “weights” to get not just qualitative results but also quantitatively accurate results. I did not try, but one could imagine choosing population shares for the Charness and Rabin (2002) “types” to match moments with reality, then using that population for other scenarios." --- To clarify, this about what research questions are likely to work well here, not what questions posed to LLMs will work well.