8 ms·

The death and life of prediction markets at Google

273 points by mfro 2y ago

sss111 2y ago
Google’s success wasn’t driven solely by its whimsical culture; the rapid growth of the market played a significant role as well!
kibwen 2y ago
I was hoping the article would reflect on the problems with predictions markets, but it's just a dry history. Crucially, "predictions markets" do not and cannot exist in any real sense. A pure predictions market would be completely isolated, causality-wise, from the event they are trying to predict. But the two are not and cannot be isolated, except for some degenerate cases like trying to guess the output of a true random number generator (and even then I'm not so sure sufficiently motivated people wouldn't try to game the system anyway). This is why we have problems with our current predictions markets, e.g. the stock market (insider trading, etc.) and sports betting (match-fixing, etc.). Every prediction with a stake is an incentive to alter the outcome of an event. Once the weight of the stake outweighs the resources being used to ensure the impartiality of the outcome, the wheels fully come off the cart and the prediction stops being about the underlying event and starts self-referentially predicting the impact of the prediction itself. The snake eats its own tail and the market becomes useless. You cannot scale up a predictions market without this eventually coming to pass. See also the famous example of how a predictions market for when public figures will die is just an assassination market with extra steps.
- cj 2y ago
  Prediction markets (like the trump vs harris bet on Robinhood) can also be used as a hedge. E.g. if before the election you think that a certain candidate winning would cause the markets to react in a certain direction, you could "bet" on the other candidate so that if your portfolio value goes down, you earn the proceeds from the bet to recoup some of your portfolio losses. Or if the "good for stock market" candidate wins, you loose the money you bet but the gains in your portfolio balances it out. In that case, you're not really betting on who you think will win. You're just betting as a hedge just in case that person wins.
  - JumpCrisscross 2y ago
    [flagged]
    - Jerrrrrrry 2y ago
      the world can be idiotic far longer than...
    - dgfitz 2y ago
      This is an idiots take.
      - Jerrrrrrry 2y ago
        It's a nuanced take, re-parse. It is literally an "idiot" put, betting on idiocy.
    - failbuffer 2y ago
      This is the Hacker News community. Let's be constructive and civil. Your comment would have more interesting and relevant if you had explained why this trading strategy is a bad idea instead of just labeling it as "idiotic".
      - Dalewyn 2y ago
        I'll bite, I guess: * In deferrence to Boglehead philosophy, hedging bets is a fool's errand because the idea is you lose your money the more you touch it. Make a plan, invest, and then hold hold hold staying the course come hell or high water. * If you truly want to reduce or eliminate risk, the best way is to simply cash out. A $1 bill will always be a $1 bill with absolute certainty.
        sodality2 2y ago
        As a boglehead... that's just not how it works in the real world. Some people would treat the loss of $X worse than the gain of $X is good. Thus, they don't have linear value for money (no one really does... if you lose 90% of your bank account, you still have dinner tonight; if you lose all of it, you might not). Some people want to come out neutral or lose a guaranteed small amount, rather than the chance to lose or gain the same amount. I'd pay $5 to avoid having to flip a $10k +/- coin. Thus, if I knew I would lose $10k if X got elected, I could place a $10k bet for Y to win.
- deleted 2y ago
  [deleted]
- ddp26 2y ago
  (Author here.) I agree with this critique in theory, but not in practice. The stakes don't need to be high to encourage strong participation. Corporate prediction markets have already scaled quite far, and those who have studied them don't find evidence of manipulation. Can't let perfect be the enemy of the good! If you want a fuller critique of prediction markets in the corporate setting, see the Dec 2021 article linked near the end [1]. [1] https://forum.effectivealtruism.org/posts/dQhjwHA7LhfE8YpYF/prediction-markets-in-the-corporate-setting https://forum.effectivealtruism.org/posts/dQhjwHA7LhfE8YpYF/...
  - joe_the_user 2y ago
    The stakes don't need to be high to encourage strong participation. OK but strong participation isn't necessarily a positive for the accuracy of a market. The problem is the prediction market with lots of participants can be just outlet for partisans to put forward their opinions. There's a tax on wrong opinions but someone is spending a bucks, the markets won't be a powerful force for changing those opinions. What you want for a prediction market is for the major participants to actively researching the problems - expend money and effort to have a well founded reason for their positions. Markets for random real-world events have the problem that many events don't occur often to weed out arbitrary biases and there may not be any easy or cost effect way to attain a well-founded opinion on the subject.
  - vlovich123 2y ago
    From your own article: > A senior executive saw Prophit give a very low probability that the company would complete the hire of a new senior executive on time (filling the position had been a quarterly objective for the past six quarters). “The betting on this goal was extremely harsh. I am shocked and outraged by the lack of brown-nosing at this company,” the executive said to laughter in a company-wide meeting. But the market was the nudge the execs needed. They subsequently “made some hard decisions” to complete the hire on time. Indeed. The whole point of the prediction markets espoused is to alter the decisions being made. That means the prediction itself can have an impact on the outcome intentionally or otherwise.
    - 2y ago
- JumpCrisscross 2y ago
  > the two are not and cannot be isolated, except for some degenerate cases like trying to guess the output of a true random number generator You’re describing endogeneity. It’s unlikely prediction markets affect natural disaster odds.
  - kibwen 2y ago
    > It’s unlikely prediction markets affect natural disaster odds. They don't need to. Predictions markets for natural disasters exist, we call them insurance companies, and the concept of moral hazard when it comes to insurance is a well-studied topic: https://en.wikipedia.org/wiki/Moral_hazard https://en.wikipedia.org/wiki/Moral_hazard
    - JumpCrisscross 2y ago
      One, insurance happens at insurance companies and in markets. Two, there are prediction markets for natural disasters. Three, moral hazard doesn’t mean insurance makes natural disasters more likely.
      - kibwen 2y ago
        The distinction matters both because the precise wording of specific contracts will cause distinctions that were not adequately considered by certain participants, and also because of who enforces the contract at a human level. Who determines what does and does not qualify as a natural disaster? If that entity has a stake in one outcome, or is influenced by an entity with a stake in one outcome, that changes the outcome. Both of these are seen in insurance claims, where homeowners find out that rising rainwater does not actually count as a "flood", and where agents of the insurance company are financially incentivized not to pay out at all if they can help it. These factors do not go away just because you take away the insurance company, you've simply diffused them.
- strken 2y ago
  What do "any real sense" and "scale" mean here? I was watching the Betfair odds during the US election, and that market alone had about $500 million dollars of bets pass through it. Is that not real? Is half a billion dollars not sufficiently scaled up?
  - mrbombastic 2y ago
    They mean that once they reach a certain scale they cease to be predictors and start affecting outcomes, not that it is impossible to allow people to bet on things indefinitely.
- mfro 2y ago
  You make a very good argument against prediction markets in which the stake has actual material value, but what about markets where the only stake is your reputation? My understanding of the Google platforms is that there were prize rewards for top predictors but largely the rewards were social, which, to me, implies they were not taken particularly seriously. An ideal platform might be one where there is no material incentive but users are instead interested in genuine hypothesizing about future events. Obviously there must be a genuine stake involved, but as we see with Manifold this can be done with 'play money'. Would be very interested to see further discussion about this.
  - kibwen 2y ago
    The two problems I identify with this are 1) it underestimates the distorting effect of social rewards (see also karma farmers and serial confabulators on Reddit) and 2) it overlooks how verifiable reputation can be turned into profit by e.g. account-selling (once again, see Reddit).
  - eitally 2y ago
    Very true. At Google it was treated much like Memegen where top bettors -- like top memers -- were rewarded with clout & notoriety, but it basically stopped there. A lot of the wagers actually were on "inside baseball" topics (will x/y/z be deprecated by a//b/c date, will perf metrics align with targets, etc). There are a few handfuls of folks who are really into it, but the vast majority of use is casual.
- s1artibartfast 2y ago
  lack of isolation isnt necessarily a insurmountable problem, as the market can still forecast, even if it is sometimes a self fulfilling prophecy. Similarly, this distortion does not necessarily overshadow the utility - e.g. the stock market may have insider trading, but is still an incredibly useful financial tool. They key thing to remember is that these markets can have utility beyond zero-sum betting aspect. For example, the stock market isnt just gamblers betting against each other, but is also a tool for auctioning corporate ownership and raising corporate funds.
- jjmarr 2y ago
  > This is why we have problems with our current predictions markets, e.g. the stock market (insider trading, etc.) A unit of stock represents a legal claim on a company's assets even in the absence of a market for that stock. In many situations, we do want people to influence the value of their stock. Any company that grants stock is doing so because they expect employees to work harder. There are many cases in which employees with stock grants might make short-sighted decisions for quick profits, but on the whole stock grants make employees into better workers. > A senior executive saw Prophit give a very low probability that the company would complete the hire of a new senior executive on time (filling the position had been a quarterly objective for the past six quarters). “The betting on this goal was extremely harsh. I am shocked and outraged by the lack of brown-nosing at this company,” the executive said to laughter in a company-wide meeting. But the market was the nudge the execs needed. They subsequently “made some hard decisions” to complete the hire on time. In this case, the predictions market would've been useless had the executive team not used it in their decision-making. A pure predictions market would be completely isolated from the event they're trying to predict. But I would say that's not an ideal predictions market.
  - adamc 2y ago
    I think evidence would make this a stronger argument. Most of the people I've known with modest stock grants ignored them, or at least that's what they said -- kind of "maybe it will be worth something, but probably not". For really large grants, I'm sure it's different, but that is a far smaller number of employees.
    - leetcrew 2y ago
      stock grants don't solve the principal agent problem at a sufficiently large company. if I have an absolutely incredible year as an IC, I might increase my employers revenue by a few basis points. I have plenty of internal metrics to demonstrate my contribution, but nothing I do measurably affects the stock price.
  - kibwen 2y ago
    The anecdote of the Google executive is a lovely example of why the market failed to work as a predictions market. If you had a stake on Google executives continuing to drag their feet on filling that position based on their past performance, and then they saw that prediction and were so incensed that they stopped dragging their feet, then you lost your stake!! The existence of the market changed the outcome; that's the whole point of the objection here! If you believe the purpose of such a market is to affect the real world in such a way, then sure, that's a reasonable stance... but that's not a market for making predictions, that's a distributed market for bidding on contracts.
- ghaff 2y ago
  I've spent some time looking at group decision making for specific situations and there are clearly some examples where they work very well--and I think you hit on one of the factors. The individual guesses don't affect the outcome and aren't being used to hedge anything. I think there are some other factors as well but I'm not sure I've ever seen a particularly persuasive framework for when prediction markets can work and when they don't.
- currymj 2y ago
  the same dynamic is true of insurance, credit default swaps, and lots of other financial products. this causes trouble from time to time but can be managed if the products serve a useful purpose.
- duped 2y ago
  How does a prediction market for say, hurricanes or earthquakes effect the outcome of those events?
  - mejutoco 2y ago
    An event like a pandemic could be not classified as a pandemic by an authority to avoid paying some insurances compensation. It is not an earthquake, but a similar example, I believe.
  - saithound 2y ago
    Prediction markets work via resolution criteria, and resolution is a human action. So the moment you operationalize any question, prediction markets can affect the outcome. To give an extreme example, when employees of the FHA bet on "Florida hurricane with 100+ casualties in 2024", they don't need to invent a weather control device to make that happen.
- __MatrixMan__ 2y ago
  I think there are cases where the outcome of the event can be bound to the prediction. Also we should be looking for cases where: > Every prediction with a stake is an incentive to alter the outcome of an event ...is a feature not a bug. For instance, imagine if PR's on core infrastructure (like xz-utils, for instance) were first reviewed by a set of trusted maintainers, and--supposing they pass that gate--were later put into a sort of limbo where people can bet on whether they'll be merged. The maintainers then make a policy around betting that the PR will be merged. They're in charge of whether it actually gets merged or not, so most of the time this is a pretty good bet and they just recycle that money by winning and then re-betting it on the next PR. Of course third parties can also bet in the same way--these would be stakeholders which are not maintainers, but who wish to "sweeten the pot" and encourage people to spend enough time with the pending code that they might find a reason to "bet against the house". No prior coordination between the maintainers and the stakeholders is necessary. Suppose I notice some malicious code in one of these commits which is in the betting phase. (Maybe I do my own testing on pre-release versions as a stakeholder, or maybe I'm a bug bounty hunter.) I can bet that the PR won't be merged. The monetary value of that bet makes it clear that even though I'm a stranger, I'm not a spammer. That "buys" me the attention of the maintainers, and if I'm right about the code being malicious, the maintainers will decide not to merge the commit: I'll have successfully altered the course of events (bad commit doesn't get merged) such that I get paid for having been right.
- mrbombastic 2y ago
  Great comment, I was fairly horrified to open Robinhood and be faced with a prediction market for the US election. I somehow missed completely that this has been legalized: https://amp.cnn.com/cnn/2024/10/02/business/appeals-court-allows-kalshi-election-betting https://amp.cnn.com/cnn/2024/10/02/business/appeals-court-al...
- spencerchubb 2y ago
  There are some ways to mitigate this. You can make the prize pool big enough that people will be motivated, but small enough that people won't try to sway the outcome. Also, prediction markets can be very useful for making predictions about competitors, because it would be rather difficult to influence what happens at a competing company.
  - TheRealNGenius 2y ago
    [dead]
- defen 2y ago
  > See also the famous example of how a predictions market for when public figures will die is just an assassination market with extra steps. Why not just write it so that non-natural causes of death don't pay out? More generally, make it so you can't wager on illegal events / outcomes, or ones where a crime materially affected the outcome. Anyway, if someone bets big on a public figure being assassinated, and then that public figure gets assassinated, it would seem like a good place to start investigating would be to look at the people who made a lot of money from that bet.
  - lupire 2y ago
    "why not just" because it destroys the utility. There were 2-3 assassination attempts on Trump this year. It would be valuable to incentive people to share knowledge of assassination vulnerability, to guide security efforts. Banning that defeats the purpose.
- skybrian 2y ago
  You could as easily say that journalism is an attempt to influence events. And it is, in a way! I think most journalists would be happy to point to situations where their reporting was read by influential people and maybe even changed outcomes. Pulitzer prizes are awarded for that sort of thing. The question is whether this is improper influence? Is it too influential, for bad reasons? I think a more reasonable critique of prediction markets is that it's guesswork laundering. We are given a number, but we don't know why that number was chosen. How can we tell whether it's justified? When markets move, there is a whole industry of people coming up with explanations of why it might have happened - more guesswork! A lucky guess can be helpful if it can be verified, but knowledge should be about more than making guesses. Sharing evidence is important.
- directevolve 2y ago
  A market can still be well calibrated even if it has a strong causal influence on the topics it’s predicting.
- aaron695 2y ago
  [dead]
- millipede 2y ago
  In other words, money is speech.
- throw_nbvc1234 2y ago
  A "good" prediction market is when it functions as an accountability market. The insider trading problem isn't an issue if the outcome is for a greater good. The problem is that money is the easiest way to ensure anonymous predictors are serious about their predictions and getting richer is not a greater good. If you were to require politicians to participate in a reputational prediction market, the aligned incentives might make it a positive thing. You could let ordinary people piggy back participate too, and then use the results to filter through internet/media noise but that starts to smell too social score-ish.
- ruuda 2y ago
  My experience from predicting on Metaculus, and following this space for a few years, is that it's hard to operationalize the thing you want to predict precisely. Regularly, things go completely sideways in ways that the author did not foresee, and the market ends up hinging on some technicality, rather than the spirit of the question it tried to predict. Some examples: - A question about asset prices specifies FTX as resolution source, but then FTX stopped existing. - There was a question about wether submarine cables in the Red Sea would be destroyed by a hostile act before a certain date. The cables got damaged, but it seemed to be a (suspicious) accident, with very limited independent media coverage. - There is a question about wether YouTube would be blocked in a certain country before 2025. It got throttled, to the point where it is unusable in practice, but not technically blocked. - There is a question trying to forecast LLM progress. How to quantify that? It chose "What is the state of the art score on the Penn Treebank at a certain date?", which was a standard benchmark at the time. But as LLMs evolved, new benchmarks got developed, and although current LLMs probably score much better on the Penn Treebank than a few years ago, nobody reports Penn Treebank score any more. The question ended up being about the popularity of the benchmark rather than LLM progress.
fnfjfk 2y ago
Google hosted multiple internal gambling platforms for employees, and HR, compliance, etc. were okay with that?
- ddp26 2y ago
  (Author here) It's complicated :-). I will say: it's hard to define "gambling" in the corporate setting. Is it gambling to do an extra project in hope of getting a "Spot Bonus" (~$250-$1000) in recognition? That was a very standardized process at Google. Is it gambling to file a patent application, which if approved, would lead to a $1000 bonus and a trophy you'd often see on the desks of Google Brain researchers? Is it gambling to decline auto-sale of RSUs, and have your compensation determined more by movements in GOOG than in your cash salary?
  - fnfjfk 2y ago
    1. No, Google isn’t taking actual money bets from their employees. 2. No, Google isn’t taking actual money bets from their employees. 3. What employees do with their vested RSUs isn’t at all the same as hosting an internal gambling platform. Once they are vested, employees own the stock, they can do whatever they want. One could sell them all and use the money to bet on roulette, which I think is obviously gambling, but that’s also obviously not Google hosting a gambling platform internally? I’m just surprised that Google hosting a platform where employees gambled was… allowed? This isn’t a moral judgement, I’m surprised that Google was operating a gambling platform internally and legal etc. thought that was a good idea.
    - s1artibartfast 2y ago
      I think you are making a lot of assumptions about how the program was run. For example, are you assuming that the employees were betting their own money or exposed to any downside at all? I assume they were given free credit for the system, and had the chance to turn it into cash bonus if they won. It is closer to a company giving out a prize for the winner of a free fantasy sports league.
s1artibartfast 2y ago
Great writeup. I think prediction markets will break through when they figure out how to capitalize not only on collective judgment, but super-forecasting. As currently formulated, prediction market outputs are just a fancy opinion poll, where participants have some incentive for accuracy. To rise above the simple wisdom of the crowds, you would want to identify the subset of market participants that are constantly beating the market (because they have a more accurate mental model of the world). I think this necessitates both 1) long term tracking of bets and 2) likely withholding individual positions from the market to prevent follower effects. Similar to the title article, this raises the question of who the ultimate customer is prediction market is. Individuals can be incentivized to bet by winnings, but who else is the customer for aggregated data? I wonder about the extent to which current prediction markets have internal outputs and derivative statistics, and what they might do with it. If polymarket or similar companies put Trump vs Harris at 55-45, do they have internal statistics that that put the race at 80-20% among their most accurate betters? Was this data for sale?
- JumpCrisscross 2y ago
  There is a conflict of interest between the information prerogative, which requires facilitating the law of large numbers and factors stability, and the profitability of gambling, which favours encouraging larger bets and volatility.
  - s1artibartfast 2y ago
    My point of curiosity is if these prerogatives can (or have been) reconciled. To that end, I'm not sure that these factors are mutually exclusive and would like to hear more of your thoughts. If I understand you correctly, I would think that a market could have both volume and depth. With respect to volatility-stability, what do you see as the drivers there? Is it that gamblers would need a significant upside to drive betting? Is this solved by the size of the bet? I suppose there is an internal conflict. If the line of a bet is 49-51% on a binary outcome, the risk of ruin is high, and the upside is low. You would need to aggregate outcomes over many distinct events to mitigate. I suppose this could hang on ability/inclination of professional forecasters to research and take several positions.
tengbretson 2y ago
> Some questions aimed to predict its competitors' next moves, such as “Will Apple launch a computer based on Intel's Power PC chip?” Is there a point spread on journalistic accuracy? How do I take the under?
- Sniffnoy 2y ago
  You should make your criticism explicit; just making a joke like this isn't particularly helpful. To make explicit what I'm assuming is your point, this statement from the article can't be correct because A. POWER is by IBM, not Intel and B. Apple had already launched such a computer back in 1994, before Google existed. Maybe x86 was meant instead?
currymj 2y ago
the market price gives you a number between 0 and 1, which should move higher as the event becomes more likely, so it's pretty useful. however interpreting it as a probability, or an average of agent beliefs, or anything like that, seems tricky. i assume these internal markets are not deep and liquid enough that you can just throw up your hands and say "EMH". it works if you assume risk-neutral traders who will just trade up to their correct price but as I understand it, breaks down with realistic traders who may limited capital and are usually somewhat risk-averse. i wonder how these prediction markets dealt with that. was there any postprocessing of market prices to get final probabilities? based on interviews with traders or observed trading behavior, did the traders behave in such a way that the market price could be interpreted as "pretty much" just a probability?
- ddp26 2y ago
  (Author here.) This is generally checked via calibration charts, e.g. bucketing markets at various points in time into 0-5%, 5-10%, etc; then counting how often the underlying events actually happen. The more they match, the more it's reasonable to interpret the market prices as probabilities. Google published [1] one such calibration chart on its current prediction market in late 2021. Also, the 2009 paper in the article [2] on Google's first prediction market published one too. [1] https://cloud.google.com/blog/topics/solutions-how-tos/design-patterns-in-googles-prediction-market-on-google-cloud https://cloud.google.com/blog/topics/solutions-how-tos/desig... [2] https://static.googleusercontent.com/media/services.google.com/en//blog_resources/google_prediction_market_paper.pdf https://static.googleusercontent.com/media/services.google.c...
  - currymj 2y ago
    thanks for the response! those calibration charts don't look too bad. that's reassuring.
- yorwba 2y ago
  Even if you assume perfectly spherical rational traders with equal fixed budgets, prediction market prices aren't guaranteed to converge to the average belief: https://quantian.substack.com/p/market-prices-are-not-probabilities https://quantian.substack.com/p/market-prices-are-not-probab... You cannot really do postprocessing to the market price to get the average belief back out, because the bounds aren't very tight: a market price of 50% could correspond to an average belief anywhere between 29% and 71%.
n8cpdx 2y ago
> Google pioneered many now standard tech practices: on-site cafés, A/B tests, and “dogfooding,” or first releasing new products internally where they can be improved before launching to the public. Famously, Microsoft and others pioneered dogfooding decades before the events described in this article and approximately a decade (at least) before Google came into existence. And I’m 99% certain company cafes existed at least a half century before Google invented the concept.
- kevindamm 2y ago
  I worked at a few tech companies before 2000 that had a company cafe but they all required payment. It was cheaper and closer but not free, and not nearly the same level of quality. Charlie's et al. were all free, and I remember even seeing a TV news spot about the free food at Google specifically, in the months before I started there. I think I'd be okay crediting Google with the free lunch (and the nod towards TANSTAAFL that I suspect it was). But dogfooding, yeah that had been around for a while. Originally from Alpo, iirc. The first tech company to adopt the term as well as the practice was Microsoft in 1988. A/B existed before Google but 2000s era A/B testing and user research were unparalleled until Facebook also started putting serious capital into it. Nowadays it's considered table stakes but it was revolutionized in the beginning of the millennium. Maybe not entirely by Google but substantially so, and driven heavily by their product launch review process.
  - lupire 2y ago
    A/B testing was Amazon Weblab. Google pioneered information retrieval and ranking innovations, not behavior optimization.
- rootusrootus 2y ago
  Definitely. I remember a class trip to Intel's Jones Farm campus back in the early 90s and they definitely had company cafes. Everything was free.
  - medler 2y ago
hanniabu 2y ago
Polymarket is the most successful prediction market imo
- beaugunderson 2y ago
  Metaculus and Manifold Markets are both "purer" alternatives
yapyap 2y ago
Google, can’t live with them, can’t do much online without them
bdjsiqoocwk 2y ago
> Yet what executives would have wanted to know was “Will Microsoft integrate LLMs into Outlook by Spring 2023?” I've said this many times: innovation at google is dead. This is yet another supporting data point. Firms that innovate aren't concerned with what competitors are doing. At google it appears it's their main concern.

273 points by mfro 2y ago

Google’s success wasn’t driven solely by its whimsical culture; the rapid growth of the market played a significant role as well!

kibwen 2y ago

I was hoping the article would reflect on the problems with predictions markets, but it's just a dry history. Crucially, "predictions markets" do not and cannot exist in any real sense. A pure predictions market would be completely isolated, causality-wise, from the event they are trying to predict. But the two are not and cannot be isolated, except for some degenerate cases like trying to guess the output of a true random number generator (and even then I'm not so sure sufficiently motivated people wouldn't try to game the system anyway). This is why we have problems with our current predictions markets, e.g. the stock market (insider trading, etc.) and sports betting (match-fixing, etc.). Every prediction with a stake is an incentive to alter the outcome of an event. Once the weight of the stake outweighs the resources being used to ensure the impartiality of the outcome, the wheels fully come off the cart and the prediction stops being about the underlying event and starts self-referentially predicting the impact of the prediction itself. The snake eats its own tail and the market becomes useless. You cannot scale up a predictions market without this eventually coming to pass. See also the famous example of how a predictions market for when public figures will die is just an assassination market with extra steps.

cj 2y ago
Prediction markets (like the trump vs harris bet on Robinhood) can also be used as a hedge. E.g. if before the election you think that a certain candidate winning would cause the markets to react in a certain direction, you could "bet" on the other candidate so that if your portfolio value goes down, you earn the proceeds from the bet to recoup some of your portfolio losses. Or if the "good for stock market" candidate wins, you loose the money you bet but the gains in your portfolio balances it out. In that case, you're not really betting on who you think will win. You're just betting as a hedge just in case that person wins.
- JumpCrisscross 2y ago
  [flagged]
  - Jerrrrrrry 2y ago
    the world can be idiotic far longer than...
  - dgfitz 2y ago
    This is an idiots take.
    - Jerrrrrrry 2y ago
      It's a nuanced take, re-parse. It is literally an "idiot" put, betting on idiocy.
  - failbuffer 2y ago
    This is the Hacker News community. Let's be constructive and civil. Your comment would have more interesting and relevant if you had explained why this trading strategy is a bad idea instead of just labeling it as "idiotic".
    - Dalewyn 2y ago
      I'll bite, I guess: * In deferrence to Boglehead philosophy, hedging bets is a fool's errand because the idea is you lose your money the more you touch it. Make a plan, invest, and then hold hold hold staying the course come hell or high water. * If you truly want to reduce or eliminate risk, the best way is to simply cash out. A $1 bill will always be a $1 bill with absolute certainty.
      - sodality2 2y ago
        As a boglehead... that's just not how it works in the real world. Some people would treat the loss of $X worse than the gain of $X is good. Thus, they don't have linear value for money (no one really does... if you lose 90% of your bank account, you still have dinner tonight; if you lose all of it, you might not). Some people want to come out neutral or lose a guaranteed small amount, rather than the chance to lose or gain the same amount. I'd pay $5 to avoid having to flip a $10k +/- coin. Thus, if I knew I would lose $10k if X got elected, I could place a $10k bet for Y to win.
deleted 2y ago
[deleted]
ddp26 2y ago
(Author here.) I agree with this critique in theory, but not in practice. The stakes don't need to be high to encourage strong participation. Corporate prediction markets have already scaled quite far, and those who have studied them don't find evidence of manipulation. Can't let perfect be the enemy of the good! If you want a fuller critique of prediction markets in the corporate setting, see the Dec 2021 article linked near the end [1]. [1] https://forum.effectivealtruism.org/posts/dQhjwHA7LhfE8YpYF/prediction-markets-in-the-corporate-setting https://forum.effectivealtruism.org/posts/dQhjwHA7LhfE8YpYF/...
- joe_the_user 2y ago
  The stakes don't need to be high to encourage strong participation. OK but strong participation isn't necessarily a positive for the accuracy of a market. The problem is the prediction market with lots of participants can be just outlet for partisans to put forward their opinions. There's a tax on wrong opinions but someone is spending a bucks, the markets won't be a powerful force for changing those opinions. What you want for a prediction market is for the major participants to actively researching the problems - expend money and effort to have a well founded reason for their positions. Markets for random real-world events have the problem that many events don't occur often to weed out arbitrary biases and there may not be any easy or cost effect way to attain a well-founded opinion on the subject.
- vlovich123 2y ago
  From your own article: > A senior executive saw Prophit give a very low probability that the company would complete the hire of a new senior executive on time (filling the position had been a quarterly objective for the past six quarters). “The betting on this goal was extremely harsh. I am shocked and outraged by the lack of brown-nosing at this company,” the executive said to laughter in a company-wide meeting. But the market was the nudge the execs needed. They subsequently “made some hard decisions” to complete the hire on time. Indeed. The whole point of the prediction markets espoused is to alter the decisions being made. That means the prediction itself can have an impact on the outcome intentionally or otherwise.
  - 2y ago
JumpCrisscross 2y ago
> the two are not and cannot be isolated, except for some degenerate cases like trying to guess the output of a true random number generator You’re describing endogeneity. It’s unlikely prediction markets affect natural disaster odds.
- kibwen 2y ago
  > It’s unlikely prediction markets affect natural disaster odds. They don't need to. Predictions markets for natural disasters exist, we call them insurance companies, and the concept of moral hazard when it comes to insurance is a well-studied topic: https://en.wikipedia.org/wiki/Moral_hazard https://en.wikipedia.org/wiki/Moral_hazard
  - JumpCrisscross 2y ago
    One, insurance happens at insurance companies and in markets. Two, there are prediction markets for natural disasters. Three, moral hazard doesn’t mean insurance makes natural disasters more likely.
    - kibwen 2y ago
      The distinction matters both because the precise wording of specific contracts will cause distinctions that were not adequately considered by certain participants, and also because of who enforces the contract at a human level. Who determines what does and does not qualify as a natural disaster? If that entity has a stake in one outcome, or is influenced by an entity with a stake in one outcome, that changes the outcome. Both of these are seen in insurance claims, where homeowners find out that rising rainwater does not actually count as a "flood", and where agents of the insurance company are financially incentivized not to pay out at all if they can help it. These factors do not go away just because you take away the insurance company, you've simply diffused them.
strken 2y ago
What do "any real sense" and "scale" mean here? I was watching the Betfair odds during the US election, and that market alone had about $500 million dollars of bets pass through it. Is that not real? Is half a billion dollars not sufficiently scaled up?
- mrbombastic 2y ago
  They mean that once they reach a certain scale they cease to be predictors and start affecting outcomes, not that it is impossible to allow people to bet on things indefinitely.
mfro 2y ago
You make a very good argument against prediction markets in which the stake has actual material value, but what about markets where the only stake is your reputation? My understanding of the Google platforms is that there were prize rewards for top predictors but largely the rewards were social, which, to me, implies they were not taken particularly seriously. An ideal platform might be one where there is no material incentive but users are instead interested in genuine hypothesizing about future events. Obviously there must be a genuine stake involved, but as we see with Manifold this can be done with 'play money'. Would be very interested to see further discussion about this.
- kibwen 2y ago
  The two problems I identify with this are 1) it underestimates the distorting effect of social rewards (see also karma farmers and serial confabulators on Reddit) and 2) it overlooks how verifiable reputation can be turned into profit by e.g. account-selling (once again, see Reddit).
- eitally 2y ago
  Very true. At Google it was treated much like Memegen where top bettors -- like top memers -- were rewarded with clout & notoriety, but it basically stopped there. A lot of the wagers actually were on "inside baseball" topics (will x/y/z be deprecated by a//b/c date, will perf metrics align with targets, etc). There are a few handfuls of folks who are really into it, but the vast majority of use is casual.
s1artibartfast 2y ago
lack of isolation isnt necessarily a insurmountable problem, as the market can still forecast, even if it is sometimes a self fulfilling prophecy. Similarly, this distortion does not necessarily overshadow the utility - e.g. the stock market may have insider trading, but is still an incredibly useful financial tool. They key thing to remember is that these markets can have utility beyond zero-sum betting aspect. For example, the stock market isnt just gamblers betting against each other, but is also a tool for auctioning corporate ownership and raising corporate funds.
jjmarr 2y ago
> This is why we have problems with our current predictions markets, e.g. the stock market (insider trading, etc.) A unit of stock represents a legal claim on a company's assets even in the absence of a market for that stock. In many situations, we do want people to influence the value of their stock. Any company that grants stock is doing so because they expect employees to work harder. There are many cases in which employees with stock grants might make short-sighted decisions for quick profits, but on the whole stock grants make employees into better workers. > A senior executive saw Prophit give a very low probability that the company would complete the hire of a new senior executive on time (filling the position had been a quarterly objective for the past six quarters). “The betting on this goal was extremely harsh. I am shocked and outraged by the lack of brown-nosing at this company,” the executive said to laughter in a company-wide meeting. But the market was the nudge the execs needed. They subsequently “made some hard decisions” to complete the hire on time. In this case, the predictions market would've been useless had the executive team not used it in their decision-making. A pure predictions market would be completely isolated from the event they're trying to predict. But I would say that's not an ideal predictions market.
- adamc 2y ago
  I think evidence would make this a stronger argument. Most of the people I've known with modest stock grants ignored them, or at least that's what they said -- kind of "maybe it will be worth something, but probably not". For really large grants, I'm sure it's different, but that is a far smaller number of employees.
  - leetcrew 2y ago
    stock grants don't solve the principal agent problem at a sufficiently large company. if I have an absolutely incredible year as an IC, I might increase my employers revenue by a few basis points. I have plenty of internal metrics to demonstrate my contribution, but nothing I do measurably affects the stock price.
- kibwen 2y ago
  The anecdote of the Google executive is a lovely example of why the market failed to work as a predictions market. If you had a stake on Google executives continuing to drag their feet on filling that position based on their past performance, and then they saw that prediction and were so incensed that they stopped dragging their feet, then you lost your stake!! The existence of the market changed the outcome; that's the whole point of the objection here! If you believe the purpose of such a market is to affect the real world in such a way, then sure, that's a reasonable stance... but that's not a market for making predictions, that's a distributed market for bidding on contracts.
ghaff 2y ago
I've spent some time looking at group decision making for specific situations and there are clearly some examples where they work very well--and I think you hit on one of the factors. The individual guesses don't affect the outcome and aren't being used to hedge anything. I think there are some other factors as well but I'm not sure I've ever seen a particularly persuasive framework for when prediction markets can work and when they don't.
currymj 2y ago
the same dynamic is true of insurance, credit default swaps, and lots of other financial products. this causes trouble from time to time but can be managed if the products serve a useful purpose.
duped 2y ago
How does a prediction market for say, hurricanes or earthquakes effect the outcome of those events?
- mejutoco 2y ago
  An event like a pandemic could be not classified as a pandemic by an authority to avoid paying some insurances compensation. It is not an earthquake, but a similar example, I believe.
- saithound 2y ago
  Prediction markets work via resolution criteria, and resolution is a human action. So the moment you operationalize any question, prediction markets can affect the outcome. To give an extreme example, when employees of the FHA bet on "Florida hurricane with 100+ casualties in 2024", they don't need to invent a weather control device to make that happen.
__MatrixMan__ 2y ago
I think there are cases where the outcome of the event can be bound to the prediction. Also we should be looking for cases where: > Every prediction with a stake is an incentive to alter the outcome of an event ...is a feature not a bug. For instance, imagine if PR's on core infrastructure (like xz-utils, for instance) were first reviewed by a set of trusted maintainers, and--supposing they pass that gate--were later put into a sort of limbo where people can bet on whether they'll be merged. The maintainers then make a policy around betting that the PR will be merged. They're in charge of whether it actually gets merged or not, so most of the time this is a pretty good bet and they just recycle that money by winning and then re-betting it on the next PR. Of course third parties can also bet in the same way--these would be stakeholders which are not maintainers, but who wish to "sweeten the pot" and encourage people to spend enough time with the pending code that they might find a reason to "bet against the house". No prior coordination between the maintainers and the stakeholders is necessary. Suppose I notice some malicious code in one of these commits which is in the betting phase. (Maybe I do my own testing on pre-release versions as a stakeholder, or maybe I'm a bug bounty hunter.) I can bet that the PR won't be merged. The monetary value of that bet makes it clear that even though I'm a stranger, I'm not a spammer. That "buys" me the attention of the maintainers, and if I'm right about the code being malicious, the maintainers will decide not to merge the commit: I'll have successfully altered the course of events (bad commit doesn't get merged) such that I get paid for having been right.
mrbombastic 2y ago
Great comment, I was fairly horrified to open Robinhood and be faced with a prediction market for the US election. I somehow missed completely that this has been legalized: https://amp.cnn.com/cnn/2024/10/02/business/appeals-court-allows-kalshi-election-betting https://amp.cnn.com/cnn/2024/10/02/business/appeals-court-al...
spencerchubb 2y ago
There are some ways to mitigate this. You can make the prize pool big enough that people will be motivated, but small enough that people won't try to sway the outcome. Also, prediction markets can be very useful for making predictions about competitors, because it would be rather difficult to influence what happens at a competing company.
- TheRealNGenius 2y ago
  [dead]
defen 2y ago
> See also the famous example of how a predictions market for when public figures will die is just an assassination market with extra steps. Why not just write it so that non-natural causes of death don't pay out? More generally, make it so you can't wager on illegal events / outcomes, or ones where a crime materially affected the outcome. Anyway, if someone bets big on a public figure being assassinated, and then that public figure gets assassinated, it would seem like a good place to start investigating would be to look at the people who made a lot of money from that bet.
- lupire 2y ago
  "why not just" because it destroys the utility. There were 2-3 assassination attempts on Trump this year. It would be valuable to incentive people to share knowledge of assassination vulnerability, to guide security efforts. Banning that defeats the purpose.
skybrian 2y ago
You could as easily say that journalism is an attempt to influence events. And it is, in a way! I think most journalists would be happy to point to situations where their reporting was read by influential people and maybe even changed outcomes. Pulitzer prizes are awarded for that sort of thing. The question is whether this is improper influence? Is it too influential, for bad reasons? I think a more reasonable critique of prediction markets is that it's guesswork laundering. We are given a number, but we don't know why that number was chosen. How can we tell whether it's justified? When markets move, there is a whole industry of people coming up with explanations of why it might have happened - more guesswork! A lucky guess can be helpful if it can be verified, but knowledge should be about more than making guesses. Sharing evidence is important.
directevolve 2y ago
A market can still be well calibrated even if it has a strong causal influence on the topics it’s predicting.
aaron695 2y ago
[dead]
millipede 2y ago
In other words, money is speech.
throw_nbvc1234 2y ago
A "good" prediction market is when it functions as an accountability market. The insider trading problem isn't an issue if the outcome is for a greater good. The problem is that money is the easiest way to ensure anonymous predictors are serious about their predictions and getting richer is not a greater good. If you were to require politicians to participate in a reputational prediction market, the aligned incentives might make it a positive thing. You could let ordinary people piggy back participate too, and then use the results to filter through internet/media noise but that starts to smell too social score-ish.
ruuda 2y ago
My experience from predicting on Metaculus, and following this space for a few years, is that it's hard to operationalize the thing you want to predict precisely. Regularly, things go completely sideways in ways that the author did not foresee, and the market ends up hinging on some technicality, rather than the spirit of the question it tried to predict. Some examples: - A question about asset prices specifies FTX as resolution source, but then FTX stopped existing. - There was a question about wether submarine cables in the Red Sea would be destroyed by a hostile act before a certain date. The cables got damaged, but it seemed to be a (suspicious) accident, with very limited independent media coverage. - There is a question about wether YouTube would be blocked in a certain country before 2025. It got throttled, to the point where it is unusable in practice, but not technically blocked. - There is a question trying to forecast LLM progress. How to quantify that? It chose "What is the state of the art score on the Penn Treebank at a certain date?", which was a standard benchmark at the time. But as LLMs evolved, new benchmarks got developed, and although current LLMs probably score much better on the Penn Treebank than a few years ago, nobody reports Penn Treebank score any more. The question ended up being about the popularity of the benchmark rather than LLM progress.

fnfjfk 2y ago

Google hosted multiple internal gambling platforms for employees, and HR, compliance, etc. were okay with that?

ddp26 2y ago
(Author here) It's complicated :-). I will say: it's hard to define "gambling" in the corporate setting. Is it gambling to do an extra project in hope of getting a "Spot Bonus" (~$250-$1000) in recognition? That was a very standardized process at Google. Is it gambling to file a patent application, which if approved, would lead to a $1000 bonus and a trophy you'd often see on the desks of Google Brain researchers? Is it gambling to decline auto-sale of RSUs, and have your compensation determined more by movements in GOOG than in your cash salary?
- fnfjfk 2y ago
  1. No, Google isn’t taking actual money bets from their employees. 2. No, Google isn’t taking actual money bets from their employees. 3. What employees do with their vested RSUs isn’t at all the same as hosting an internal gambling platform. Once they are vested, employees own the stock, they can do whatever they want. One could sell them all and use the money to bet on roulette, which I think is obviously gambling, but that’s also obviously not Google hosting a gambling platform internally? I’m just surprised that Google hosting a platform where employees gambled was… allowed? This isn’t a moral judgement, I’m surprised that Google was operating a gambling platform internally and legal etc. thought that was a good idea.
  - s1artibartfast 2y ago
    I think you are making a lot of assumptions about how the program was run. For example, are you assuming that the employees were betting their own money or exposed to any downside at all? I assume they were given free credit for the system, and had the chance to turn it into cash bonus if they won. It is closer to a company giving out a prize for the winner of a free fantasy sports league.

s1artibartfast 2y ago

Great writeup. I think prediction markets will break through when they figure out how to capitalize not only on collective judgment, but super-forecasting. As currently formulated, prediction market outputs are just a fancy opinion poll, where participants have some incentive for accuracy. To rise above the simple wisdom of the crowds, you would want to identify the subset of market participants that are constantly beating the market (because they have a more accurate mental model of the world). I think this necessitates both 1) long term tracking of bets and 2) likely withholding individual positions from the market to prevent follower effects. Similar to the title article, this raises the question of who the ultimate customer is prediction market is. Individuals can be incentivized to bet by winnings, but who else is the customer for aggregated data? I wonder about the extent to which current prediction markets have internal outputs and derivative statistics, and what they might do with it. If polymarket or similar companies put Trump vs Harris at 55-45, do they have internal statistics that that put the race at 80-20% among their most accurate betters? Was this data for sale?

JumpCrisscross 2y ago
There is a conflict of interest between the information prerogative, which requires facilitating the law of large numbers and factors stability, and the profitability of gambling, which favours encouraging larger bets and volatility.
- s1artibartfast 2y ago
  My point of curiosity is if these prerogatives can (or have been) reconciled. To that end, I'm not sure that these factors are mutually exclusive and would like to hear more of your thoughts. If I understand you correctly, I would think that a market could have both volume and depth. With respect to volatility-stability, what do you see as the drivers there? Is it that gamblers would need a significant upside to drive betting? Is this solved by the size of the bet? I suppose there is an internal conflict. If the line of a bet is 49-51% on a binary outcome, the risk of ruin is high, and the upside is low. You would need to aggregate outcomes over many distinct events to mitigate. I suppose this could hang on ability/inclination of professional forecasters to research and take several positions.

tengbretson 2y ago

> Some questions aimed to predict its competitors' next moves, such as “Will Apple launch a computer based on Intel's Power PC chip?” Is there a point spread on journalistic accuracy? How do I take the under?

Sniffnoy 2y ago
You should make your criticism explicit; just making a joke like this isn't particularly helpful. To make explicit what I'm assuming is your point, this statement from the article can't be correct because A. POWER is by IBM, not Intel and B. Apple had already launched such a computer back in 1994, before Google existed. Maybe x86 was meant instead?

currymj 2y ago

the market price gives you a number between 0 and 1, which should move higher as the event becomes more likely, so it's pretty useful. however interpreting it as a probability, or an average of agent beliefs, or anything like that, seems tricky. i assume these internal markets are not deep and liquid enough that you can just throw up your hands and say "EMH". it works if you assume risk-neutral traders who will just trade up to their correct price but as I understand it, breaks down with realistic traders who may limited capital and are usually somewhat risk-averse. i wonder how these prediction markets dealt with that. was there any postprocessing of market prices to get final probabilities? based on interviews with traders or observed trading behavior, did the traders behave in such a way that the market price could be interpreted as "pretty much" just a probability?

ddp26 2y ago
(Author here.) This is generally checked via calibration charts, e.g. bucketing markets at various points in time into 0-5%, 5-10%, etc; then counting how often the underlying events actually happen. The more they match, the more it's reasonable to interpret the market prices as probabilities. Google published [1] one such calibration chart on its current prediction market in late 2021. Also, the 2009 paper in the article [2] on Google's first prediction market published one too. [1] https://cloud.google.com/blog/topics/solutions-how-tos/design-patterns-in-googles-prediction-market-on-google-cloud https://cloud.google.com/blog/topics/solutions-how-tos/desig... [2] https://static.googleusercontent.com/media/services.google.com/en//blog_resources/google_prediction_market_paper.pdf https://static.googleusercontent.com/media/services.google.c...
- currymj 2y ago
  thanks for the response! those calibration charts don't look too bad. that's reassuring.
yorwba 2y ago
Even if you assume perfectly spherical rational traders with equal fixed budgets, prediction market prices aren't guaranteed to converge to the average belief: https://quantian.substack.com/p/market-prices-are-not-probabilities https://quantian.substack.com/p/market-prices-are-not-probab... You cannot really do postprocessing to the market price to get the average belief back out, because the bounds aren't very tight: a market price of 50% could correspond to an average belief anywhere between 29% and 71%.

n8cpdx 2y ago

> Google pioneered many now standard tech practices: on-site cafés, A/B tests, and “dogfooding,” or first releasing new products internally where they can be improved before launching to the public. Famously, Microsoft and others pioneered dogfooding decades before the events described in this article and approximately a decade (at least) before Google came into existence. And I’m 99% certain company cafes existed at least a half century before Google invented the concept.

kevindamm 2y ago
I worked at a few tech companies before 2000 that had a company cafe but they all required payment. It was cheaper and closer but not free, and not nearly the same level of quality. Charlie's et al. were all free, and I remember even seeing a TV news spot about the free food at Google specifically, in the months before I started there. I think I'd be okay crediting Google with the free lunch (and the nod towards TANSTAAFL that I suspect it was). But dogfooding, yeah that had been around for a while. Originally from Alpo, iirc. The first tech company to adopt the term as well as the practice was Microsoft in 1988. A/B existed before Google but 2000s era A/B testing and user research were unparalleled until Facebook also started putting serious capital into it. Nowadays it's considered table stakes but it was revolutionized in the beginning of the millennium. Maybe not entirely by Google but substantially so, and driven heavily by their product launch review process.
- lupire 2y ago
  A/B testing was Amazon Weblab. Google pioneered information retrieval and ranking innovations, not behavior optimization.
rootusrootus 2y ago
Definitely. I remember a class trip to Intel's Jones Farm campus back in the early 90s and they definitely had company cafes. Everything was free.
- medler 2y ago

hanniabu 2y ago

Polymarket is the most successful prediction market imo

beaugunderson 2y ago
Metaculus and Manifold Markets are both "purer" alternatives

yapyap 2y ago

Google, can’t live with them, can’t do much online without them

bdjsiqoocwk 2y ago

> Yet what executives would have wanted to know was “Will Microsoft integrate LLMs into Outlook by Spring 2023?” I've said this many times: innovation at google is dead. This is yet another supporting data point. Firms that innovate aren't concerned with what competitors are doing. At google it appears it's their main concern.