Robin:

Hello, Agnes and Arnold.

Agnes:

Hi, Robin.

Arnold:

Hey, Robin.

Robin:

Agnes had an idea for what to talk about today. Or Arnold had the idea.

Arnold:

We wanted to talk to you about prediction markets. And in particular, because of this recent story that... Everybody's following the election, slightly obsessively, has been following PolyMarket. Because PolyMarket is a prediction market that uses real money. It's illegal. A lot of it. It's still used by many Americans. There's around a billion dollars, I think, that's been bet on this election. which is really a remarkably enormous amount of money. And it, along with the polls, which people like Nate Silver and the national polling averages and everything, have been treating this race as a kind of toss up. Everything's sort of 50-50, nearly. The Electoral College has been around 50-50. And for a long time, Polly Market was about 50-52. And then quite recently, Polly Market had a pretty dramatic shift towards Trump. It's now... Trump is at around $0.60, and Kamala Harris is at around $0.40 to win the election. And a lot of people, especially people who really don't want Trump to win, are quite dismayed by this and are trying to understand why this has happened. And there was a recent Wall Street Journal article, I think Nate Silver also pointed this out on his blog, that this recent shift is substantially due to a small number of people placing very large bets. Wall Street Journal said four accounts placing a series of $30 million bets, all on the Trump side, and that that has driven the market in one direction. We thought this would be a good opportunity to talk to you about the informational value of prediction markets as against the criticism that people often have and that people might have in this case, that these markets are manipulatable by placing large bets in order to try to get them to go in one direction or another. And you are on the record saying that they're not really manipulatable, that if people try to manipulate markets by placing large bets, that actually just makes them better sources of information. And so we'd like to talk to you about that and ask you some sort of more particular questions.

Robin:

So to clear some ground here, let me note that I'm often comparing prediction markets to many other alternative institutions. And here we're just, I guess, talking about polls or pundits as alternatives. Yeah. And that I'm especially interested in prediction markets for other applications. That is the biggest win is maybe on a topic where there just is no reasonable other source that you might use. Like say a company with a project with a deadline, you want to know what the chance of making the deadline is, then, you know, this just becomes the only reasonably well calibrated informed source you might have, and there's a huge value there. And I'm especially interested in doing that for places where you have key decisions, where the market could inform the decision. And that's also promising in corporate environments for hiring or deadlines. It would be promising here in elections if we have markets on the consequences of who we vote for, that would be the huge social value, but we don't really do that much. We have these markets on who will win, but in some sense, there's not actually that much social value. in anticipating who will win, especially since there's just two main candidates. Look, if there were three main candidates, there'd be this question of, if I vote for the wrong candidate, I'm throwing away my vote because it'll all come down to the final two. And so then there would be still some value in knowing who are the final two, and prediction markets could help with that. But here, since you only have the two main candidates, there isn't actually that much value in knowing who's likely to win. You should just pick who's bettered for you and vote for them, regardless of their relative chances. But because this is popular here, then this gives us an opportunity to talk about it. I also mentioned that roughly a century ago, or around 1900, 1920, in the United States, there was more money bet on presidential elections than on stock markets at the time. So stock markets instead have become vastly larger, and prediction markets, these betting markets, went away because of the introduction of scientific polling and the prestige of polling displaced betting markets for a long time, at least. And now in recent decades, maybe there's a return more to the prestige of betting markets. But now, as you can tell, they're still competing with polling for prestige. And that's part of the argument here.

Agnes:

Well, maybe we could start with or get prestige, what do you think is a better source of information, the polls or the prediction markets? I mean, we're specifically talking about... Conceptually, right.

Robin:

Conceptually, I think you should, in all forecasting situations, distinguish between the data that you're going to base a forecast on, and then some effort to forecast using the data. So statisticians quite commonly take a dataset and they make a statistical model and they use that model to make a forecast. And then, you know, typically you're invited to look at the statistical models forecast, not directly at the data. The data is a source. for building the models. Now, sometimes, in some sense, the statistical forecast could itself be data for some meta-statistical analysis. So there are, you know, often meta-analyses of academic fields where they take a bunch of papers, each of which had a statistical estimate, and try to do further estimates of the many different papers that they combine But I think there's this important distinction between data and forecasts. And then I would further make a distinction between a forecast that uses some method and a forum in which forecasts are disputed. So in academia, there's a difference between a particular paper, which makes an argument and makes a claim based on some data analysis and puts it forward, and then a journal or a field which takes many different papers and tries to make an overall judgment about an overall sense of the evidence at the moment. And so now I've given you three levels of analysis. There's data, there's forecasts using a method, and then there's a forum in which forecasts are disputed.

Arnold:

So can I break that down a little bit in the relevant case? So we're not polling people on who they think will win. If we were, that would be much more like a prediction market. we're polling people on how they're going to vote, and then we need to interpret that, the results of those polls, into a prediction, which is something Nate Silver does. So Nate Silver, more like the academic paper based on the evidence, where the data is just the polls. And then the prediction market is a bunch of amateur Nate Silvers. We're getting a... Some are amateur, some are professional, that's the point. Many can contribute. I'm not sure there's any professional Nate Silver other than Nate Silver, or at least very few. But the prediction market is the equivalent of academia here, right?

Robin:

Rather than- Right. A forum. Like a courtroom is a place where you decide the guilt, but the different sides will prevent their evidence and their arguments based on the evidence. And there's those three levels in a courtroom too.

Arnold:

So, maybe one lesson to take away from this, where I'm sitting at my computer, and in moments of weakness, I'm doing things like looking at some polls, and then I'll go to Nate Silver's blog, and then I'll go to Polymarket. And in each of these cases, what I should understand is that I'm not looking at three comparable sources of information. I'm looking at three very, very different sort of stages.

Robin:

decide not to go to the restaurant when you see the wilted lettuce that's coming in the back of the restaurant, fine. But they are three different steps in the process.

Agnes:

Can I ask, actually, I want to go back for a second to the idea that there's no social value or less, not that much social value to these polls. I wonder whether you think that, um, Finding out that the election is very even might have social value in that it might motivate people who are on the margin to vote. That is, is that actually a good form of reasoning? Should I feel like my vote matters more in a very close election than in one that isn't?

Robin:

So a standard policy that people often have is that they don't want media to report on exit polls until the polls have closed. And that's because they're trying to prevent people from getting information on who's likely to win under the idea that the exit polls might make you believe one's idea is likely to win and therefore not be willing to vote at later in the day. So people that have that norm, sometimes we want to prevent people from getting information about who's likely to win. And you might at some point want to do that with prediction markets, say, no, we don't want these markets telling people who's likely to win because... So that's a negative social value really.

Agnes:

No, no, no, that's not what I was saying. You misunderstood me. I was saying finding out that the election is very close. In this case, Polly Market doesn't say that, but assume it did. It did for a while. Finding that out could motivate me to vote. So that information would make me more motivated to vote because I think it's pretty close.

Robin:

Right, but when we calculate the value of information, what we have to do is average over all the different pieces of information you could get. It's no fair to pick one particular realized piece of information to calculate the value of that unless you average in the others. So you see, if you don't know if the election is close, the question is, should we tell you? If it happens to be close and we tell you, then you're more motivated. If it happens not to be close and we tell you, you won't be. We have to average over those to ask whether it's a good idea to tell you if the election's close.

Agnes:

Right.

Robin:

Right, just a key thing to admit is that this is a technology for generating and aggregating information. There are cases in the world where we don't want information aggregated and distributed, in which case you don't wanna be using these. I'm not gonna argue that every possible case, one should always get the maximum information and distribute it as maximally far as possible, right? There are just many cases where information is valuable and this is useful for those. But in the case of election day, it's quite plausible that it's bad to tell everybody how close the election is, because on average, some of the time it won't be close and you'll discourage them from voting and that creates strategic games to play earlier in the day.

Agnes:

Okay, we can go back now to manipulation.

Arnold:

So Agnes actually came up with this scenario, which is I think a good way of putting it. So we can imagine a prediction market being manipulated. For example, suppose those big bets were all placed on Trump. by a Kamala Harris supporter because this Kamala Harris supporter just happens to know that as long as Trump is winning in poly market, that'll boost turnout by 10%. That's an absurd claim, but that's just- Not crazy. And so they think that the higher the chances that poly market is giving Trump, the more likely Kamala Harris is to win. right? And so somebody betting on Trump in order to try to make Trump's polymarket chances as high as possible so that Kamala Harris will win because of the increased turnout is manipulating that market, right? They're not feeding their guess as to the election in the market. They're guessing that Kamala Harris is going to win. but they're using the market as a kind of tool to produce a certain kind of deceptive information. And people have this worry. I mean, I've talked about prediction markets with lots of people, and they very frequently have this worry that some rich person is going to just throw a ton of money at the market, determine the course of it, and then get whatever they want informationally out of it, and maybe manipulate it in their favor in one way or another. So why are markets robust against this kind of manipulation?

Agnes:

Wait, and before you answer, I want to add a thing in there, which is suppose that some rich person has done this, and so the market is now strongly in favor of And the thought would be, look, it might be that there are some people with competing information who are then going to bet against it, but there's many people who are going to see that and then just infer that that's going to be a piece of information itself, a really strong piece of information, right? And so we're all going to be misled because we're going to read things off of the market. And in fact, that will disincline us to bet against it.

Robin:

Okay, first to set context, we're considering prediction markets in the context of the world that typically uses many other social mechanisms to aggregate, to induce people to elicit and aggregate information into consensus forecasts that then other people rely on. So we have academia, we have news media, we have rumor mills. So the standard we should hold prediction markets to is by comparison with these other mechanisms. we should ask if we add this into the mix, how will that change our menu of options for different sources of information in terms of whatever the goals we want from them, but one might be accuracy. If at least we think when information is good, we would say, you know, how accurate are these different sources? And then one possible source of accuracy differences might be manipulation. We are sure that all of these other institutions, in fact, do have a lot of manipulation attempts. That is, pundits definitely write columns where they try to change perceptions of who's going to win the election. Even people pay pollsters to do polls that are slanted toward one side or the other. Rumor mills, certainly people try to pass along rumors in order to induce a perception of one side or the other. So we're in the context where manipulation is widespread across all the institutions that we're familiar with trying to produce these things. And in all these other institutions, there's a wide range of inequality of influence. So say gossip might be very egalitarian, but perhaps pundits are pretty concentrated. or major polling organizations are pretty concentrated, so.

Agnes:

I'm sorry, I'm sorry, Robin, you have to stop. You have to stop. I can't, I just can't deal with this anymore. You have a sock in your pocket, and I just, I keep trying to make myself not laugh at the sock in your pocket, but I can't. Well, it's actually a glove. Oh, okay, it looked like a sock. Okay, I'm sorry.

Robin:

There's two of them there, two gloves, but he's been there for a very long time. That's perfectly, Gloves in a coat pocket, yes. I hate it.

Agnes:

Okay, I hope we can just edit that out.

Robin:

I think our viewers will enjoy that.

Agnes:

Okay, all right, fine, keep going.

Robin:

Just overall positive value. Very sort of personal interaction, sure.

Agnes:

I was really trying. I was like trying not to keep writing the word sock down in my notes.

Robin:

Okay, well, you didn't quite succeed.

Agnes:

Yeah.

Robin:

Do we have any other further clothing disclosures we'd like to make here? Maybe we could admit that our clothes were taken from indigenous people who once had the clothes here before or something like that.

Agnes:

I missed the beginning of this sentence because I was just napping too much. Back to prediction markets.

Robin:

All right. So anyway, the point was we're comparing prediction markets to these other mechanisms and they already have lots of manipulation. And that's partly why we care about who is selected for those institutions, who we listen to as pundits, who we allow to do polls, et cetera. And you might imagine even regulating these other institutions on the basis of concern about those things. But we have a general, in the US, free speech presumption maybe that if there are competing sources and some of them may be more open to manipulation, that we would rather they suffer a reputation cost for that problem and that we switch to looking at other sources based on that being one of our concerns. that even if prediction markets were seriously subject to manipulation problems, you still might say, well, they should still be allowed as part of the mix of sources that we are allowed to consider. And that can be one of the reasons why you might discount it compared to others, but it's not necessarily a reason to ban it or close it down, per se. unless you think people are over-relying on it, but that's an issue with all the other institutions. You know, you might ban pundits if you thought we were over-relying on them or something.

Agnes:

But anyway, but so that. We weren't at all coming from a point. should we ban the production?

Robin:

No, no, right, but that's... I believe that... Well, they are, in fact, banned. You see, that's part of the context here. Sorry? They are, in fact, banned, but polymarket is... You know, if that wasn't the question, you're out. All right, I get it. Okay, right. So now we get to the manipulation question more directly. And these are financial markets and standard financial markets have some standard mathematical, you know, game theory models of their behavior. And our standard model of financial markets has two kinds of participants. There are what I'll call wolves and sheep. The wolves are what we call informed traders. We model these traders as traders who know something. And they're looking at the price and they're asking whether their information suggests that the price should be different and will eventually be different on the basis of their information and whether they have a proper opportunity in making that trade. And that's the wolf or informed trader as part of the model. And in that model, they don't trade if they don't have some information and they trade more when they have more information and when the market they think is more wrong, they will trade. So in that model, if you have a market of only wolves, they don't want to trade because they each know that While they have a piece of information suggesting one direction, in a trade they will trade with somebody in the opposite direction. And if that person is also basing their trade on information that leads in the opposite direction, then neither of them can expect a profit. So markets with wolves, nothing should really happen because they'll each be afraid to trade with other wolves. Thankfully, though, we have this other class of trader in the standard model, sheep, and they're called affectionately noise traders. And the key description of a noise trader is they trade for some other reason, God knows what, and they just make trades even though they're going to lose on average. So one example is people who buy stocks, I'd say they're saving for retirement. So every month they save a little more. So they buy whatever is available that month. And then when they retire, they start selling and they need to buy and sell, but they don't have any particular information, but because they have what's called liquidity needs, then they buy and sell. And they're the noise traders. There's somebody who trades for some other reason than having some information. So this is our standard model of financial markets, sheep and wolves. And in this model, what we have is we do have the wolves trading and the prices do become informative because they do profit by trading against the sheep. And then what we see is that the more sheep there are, the more wolves are attracted and the more accurate the prices get. There's no sheep, no wolf wants to bother, there's no profit to be made, more sheep, more profit to be made by trading against them, more wolves are attracted and wolves put in more effort to trade against the sheep. So this is just our standard model of financial markets. So this says that markets with more sheep, say larger companies, have more accurate prices, or say in elections, bigger election markets like presidential elections will have more accurate prices, they say Senate races. And this is in fact, what we see more accurate prices or the bigger markets where there are more sheep who have other reasons to trade. So notice that the sheep, each sheep is actually adding noise. That's the called noise trader, right? So holding everything else constant, if you let a noise trader in, they'll just going to knock it in a random direction away from where else it might've been. And that's not good for accuracy. So. If you just hold everything else constant and look at any one noise trader, you say, why are we letting noise traders in here? That's a bad idea. And in fact, there was a set of experiments where they had, by the good judgment people, where they had a bunch of markets with lots of people in them, and then the people who were the most accurate in those markets, they had a whole separate market for just those people. And then those markets were, in fact, more accurate. They had a way to incentivize them. But it is true that You know, if you could only have the most accurate traders and just have them traded with each other, then that market's more accurate. So there's a literal sense of which noise is making things worse. But in financial markets, noise is the reward for everyone else. And if there's no noise traders, there's no reward. So there's no everybody else. Okay. So this is why it's good to have more noise in financial markets. because that's the payoff for all the others to profit by trading against them. So I'll pause here. And if you want to push on this, because I will, the, the application of manipulation is really simple and straightforward, but let's pause at this point. Is this closing or?

Arnold:

No, I, I, I, that is, I, I, I take it that the thought is if all we have are wolves, and they're roughly comparably informed wolves, then we shouldn't see the price move at all. And because the price is going to move so little, no wolf has much reason to be trading in this market.

Robin:

What's worse, there's just not even gonna be a price. No one will trade, so there won't be a price. Just the market doesn't exist. I mean, it's important to notice the vast majority of possible betting markets don't exist. The existence of a market is an unusual thing to be explained by some unusual circumstance.

Arnold:

And then in order for there to be some reason to trade, there has to be some prospect of profit. And there's a prospect of profit when the market goes in a direction that you, as a wolf, think is contrary to the information you have. That is, something becomes overvalued. or something becomes undervalued, and you can then make a profit by trading for or against it.

Agnes:

Wait, I'm sorry. Can I ask about that? Because say it goes contrary to the direction that you expect, but say you think that it's the wolves who did that, then shouldn't you just think they have information that you don't?

Robin:

Right. That's what I was highlighting in my analysis. You think you know something, and here's a price, but what do other people know that you don't? And a key point is that in this trade, somebody else will be on the opposite side from you. So they're going to be pushing the price down as you're pushing it up. So you have to wonder, well, why do I think I know better than that other person pushing the other way? Now, if that other person is random, you see, and knows nothing, well now my little piece of information I can trust more on average that, you know, sure lots of people know other things, but as long as my new thing is like not that correlated with everything else everybody knows, probably my thing is actually relevant and I should trade on it. But I will trade with somebody who is going the other way.

Arnold:

And if you're trading in one of these markets, you want to be trading against a sheep. And so you can only, kill up non-profits to the extent that you are counting on the idea that the person on the other end of your trade is a sheep. And so you need lots of sheep in a market in order to be reasonable in that belief that you're trading. And I take it that the next step is, it doesn't particularly matter what's motivating sheep or what's going on with them. They're just kicking the ball around. It's going in random directions. And if one of them is trying to make Kamala Harris win or one of them is trying to make Donald Trump win, one of them wants the polymarket number to be high for one candidate or another, and they're trying to manipulate the market. They're just being a normal sheep. That is, there isn't even any interesting distinction between them and somebody who has Harris vibes and so is betting on Kamala Harris or Trump vibes.

Robin:

Exactly. The defining characteristic of a sheep is their motivation is something other than information they have about the value of the asset. You don't really care what motivation is. It's just some other motivation and manipulators, their motivation is in that class of other motivations.

Agnes:

Oh, I have a question. Couldn't you have a situation where the market is just mostly sheep? There's just so many sheep and there's not any wolves.

Robin:

But the number of wolves is endogenous. That is, the wolves are attracted then. The more there's this huge market with these crazy prices, the wolves go, wow, this is great. And they jump in, lots of them, and each one will make large trades. Because each wolf can decide how big of a trade to make as well.

Agnes:

Right. But in a situation where, for example, this market is illegal and there might be special circumstances, a bar that you have to pass in order to enter it, it could be that that bar is going to filter, say, for sheep.

Robin:

Right. But generally, the wolves are just more strongly motivated and they will try harder to overcome whatever obstacles you have. Because they want to eat some sheep. The sheep just want to eat some grass or something. Come on. If there's a fence, who's going to jump over the fence harder? Is it going to be the sheep to go get the grass or the wolf to go eat the sheep?

Arnold:

This does lead me to a question. In something like a presidential production market, The sheep sound, I mean, so as you pointed out, in other markets, sometimes people buy things just because they need them for whatever reason, right? They're consumers. They're not traders. And in prediction markets, there are no consumers.

Robin:

everyone has to be thinking that they're a wolf. There are a few, but it's a small contribution. So there are some businesses where basically who wins the election will affect the demand for their product or the cost of their services or regulation. And so they have a small desire to hedge the risk of the election in an election market. And there are some traders who do that, but most of the traders are not that way.

Arnold:

If somebody thinks they're going to do better in a Trump administration, they have reasons to bet on Harris. Right. Hedges their bet. Okay. But given that, are prediction markets predicated on a bunch of idiots?

Robin:

Well, first let's admit that the familiar financial markets you're aware of, like stock markets, commodity markets, options markets, currency markets, They are in fact full of sheep who are not because they need to make the trades. They are in fact enjoying the fun of speculation. So most of the money the hedge funds make, they make against trading against these other fools in the market and they're The stereotype would be somebody who became a dentist recently, they've had all these years of expenses and living poorly, and finally they're making a lot of money as a dentist. And now they feel like they're a really important, powerful person in the world. Now they want to go win at some other game and they're going to go play the stock market. And then they lose $50,000 in their first year and cut back a lot. But that's where most of the money from most financial markets comes from. So we are, all the financial markets are, in fact, predating on those people. That's the fact of familiar financial markets. So other betting markets, you know, are similarly. Now, so you might think you're predating on fools, but there is this idea that they're having fun. They're trying to prove themselves. They enjoy the sport of the game and the chase, but they are losing money.

Arnold:

And the wolf-sheep distinction, I take it that at least a fair number of sheep are people who are trying hard to be wolves but are very bad.

Agnes:

or believes they are... Right, right.

Robin:

That is the information story. Right, they may think of this as a training process or they're trying to learn how to do to eventually become a wolf. Yeah, so this is like many people hope to become athletes, hope to become musicians, actors, and a lot of those people end up doing a lot of work really cheap. in the service of that and most of them fail. And we could be in some sense exploiting all of the wannabe athletes through their hope of eventually being a successful athlete and then getting all their cheap entertainment and effort out of them early in their career because the vast majority will fail. But they seem to have fun. They seem to be highly motivated, so I'm not sure we want to stop them, but maybe we shouldn't subsidize the whole thing anyway, but not sure we should ban it. But anyway, that's the context for prediction markets. They are much like financial markets and betting markets in the sense that most of the participants are not that knowledgeable. Most of them are there for the pride, the fun, it's almost like bar bets, the bragging rights. They want to enter a contest and test their mettle and hope to prove themselves among the crowd. Most, of course, don't.

Agnes:

So, but you could have a prediction market that was all wolves if you incentivized.

Robin:

Exactly. And that, for the best information, that might in fact be the best policy. That is, we should, I have a mechanism I invented years ago for subsidizing trading in financial markets, basically a automated market maker, and that doesn't add noise. It subsidized the trading without any noise, and so there's a sense in which if you give a strong subsidy for an automated market maker and you let only the wolves trade there, you'll get the most accurate prices. Could you say a little more about that? Why doesn't it add noise? So an automated market maker, a simple mechanism is it always has a relationship between its inventory and its price, say a simple monotonic function. And therefore it's always available to buy or sell. And the price will depend on the quantity, but for very tiny amounts, the prices are very close to each other, a very small bid-ask spread. And basically anybody can walk up to this market maker and make a purchase, which will then change its quantity, which therefore changes its price. But it's such that if you did the opposite trade a moment later, somebody else did, it would move exactly back to the same price in inventory. So it doesn't make or lose money in cycles. It loses money as it moves from its initial place to some final resting place that's different. And the amount of money it loses is exactly the amount of money the other traders gain. and that's the reward being offered to the wolves to come trade against the market maker.

Arnold:

So the person who wants to produce the market just pays a penalty.

Robin:

Right. You can calculate a maximum penalty that's associated with the liquidity, the thickness of the market maker, and so it's relatively easy to just figure out how much at the maximum you might lose through creating this market maker, but you'll never lose to anybody who does anything but in fact make the market more accurate. So it's a nice guarantee on who you're paying.

Agnes:

So you've sort of noted that people seem to want to do the wolf sheep kind of prediction market more than they want to do your kind. And that in particular, they seem to want to do the wolf sheep ones for things like who will win the election or who will win the Academy Award or whatever, or there's not much social value to the prediction. Of course, related, because only if there is social value would there be an incentive to set up the market. And so why is it that we live in a world where we're drawn to worse ways of getting worse information instead of better ways of getting better information?

Robin:

Well, it's that the sheep volunteer. That is, we don't have to regulate them and force them to be sheep and subsidize all those things. They're just, out of the goodness of their heart or their own bravado and arrogance, they're subsidizing all these markets for free. And given that all those subsidies are there, people haven't felt very inclined to add more to, you know, get answers on the topics where these people are willing to do it for free. So in a sense, you might say, we are happy to see the sheep subsidize the market and happy to take the information the markets generate, but we haven't collectively decided that information is so valuable we're willing to pay for it.

Agnes:

Take gambling. Gambling would be like an example of this more broadly conceived, right? Like horse racing or dog fighting or casinos or whatever. All of those three examples involve institutions like you had to set up the horse race. I mean, you have to build the track. It's not just like people sitting around. It's like there's a

Robin:

But because the people willing to gamble on the horse race are there willing to pay, then that creates an incentive for the track to be created and to charge people that are at the track and to be there to take their, those are customers. So they've been willing to become customers and pay for things. So a lot of the ways people gamble don't create these prices that are valuable for other people. It's only some betting markets where that happens. And so notice, At least in the betting markets, we are getting some other social value from their gambling activity. Whereas say for slot machines, there's very little other value information that comes out of the slot machine room. Those people walk in with money, they walk out with nothing and the rest of us get nothing. So at least you might say we found some places where those people can gamble and their suppliers can get the money from them and the rest of us get something. Stock market is virtuous to that extent by comparison to a slot machine casino. but maybe not as virtuous as if we would just pony up and pay for the information that we've decided is valuable.

Agnes:

But in general, when you say pony up and pay, I mean, the reason we pay for things is because we already had some need that that information will fulfill. And presumably your point is that the need is already there. So it looks like customers are already there.

Robin:

Well, there's two key kinds of customers here. And in fact, this is the, in my mind, the key idea of prediction markets, the reason why there's a new thing that's interesting in the world. So for centuries, we've had betting markets where people have bet and prices were created and other people could act using the information of those prices. The new thing in the world is to say, hey, you could do this on purpose. If you had a question you wanted the answer to, you could make a market on your question, you could subsidize it and then get the answer. by paying the participants to participate through the subsidy in the market, that's a thing you could do. And the whole new thing is to say, let's try to do that more. That's the new idea in the world that I was instrumental in spreading to the world three, five years ago. And that's the whole new thing. Yes. So the world has yet to fully appreciate that it has needs. that this could satisfy and that it should be more willing to buy this product that satisfies its needs.

Agnes:

It's just very surprising to me that the world has more trouble realizing that it has informational needs than the trouble it should have realizing that it has gambling needs, which like, you know, like people who start to play online poker or something, they might not have looked inside their souls and been like, I have a need to, gamble before that thing was present before that doesn't seem very obvious to me. Whereas the informational need actually seems pretty obvious. It seems pretty clear that there's stuff where you'd benefit from knowing it. So why is it that we have easier time introspecting the gambling need relative to the informational need?

Robin:

Excellent. So, as you know, I'm author of this book, The Elephant in the Brain, co-author, Hidden Motives in Everyday Life, and this was a key piece of information to me in my life. to see that we had this plausible, easily understood, widely accepted rationale for why people should be interested in prediction markets, and then to see that they didn't seem to be very interested. This was a key data point about the differences between what we say we want and what we actually want. which I explored in many other contexts in this book. This book doesn't actually talk about prediction markets, but I'd say, yes, a key fact about the social world is that we often talk as if we had certain motives and certain priorities. And then our actions, if you look carefully, seem to belie that, that is, they don't choose the things we would choose if we had the priorities we say we have.

Agnes:

In that case, which is which? Is the information something that we only say we want, but we don't actually want? Or is it something we actually do want?

Robin:

The primary target that I've initially suggested, which I do think makes sense, is that the places in the world where there's the most value of information are organizations making key organizational choices like who to put in charge of a project or what the deadline should be or what the requirements should be or who to hire to different positions or what price to set products or what products to create and distribute in what areas. Those are the highest value decisions in our world. And so those are the most obvious places where by offering them in a process to get better information about the decisions, then more value would be realized. So that is the obvious place to look. And when you look at organizations with important decisions, you will certainly see people talking about how important those decisions are, and they will often give you sort of a scientific manager spreadsheet calculator sort of description of what they are and what they're doing. They're trying to decide whether to open this factory now or close it down or do that. And they're calculating the costs and they're finding options and they're doing robustness estimates of what could go wrong. And they present themselves as collecting information to make key decisions. And given that presentation, they should, of course, be very interested in further information. And yet, in fact, they aren't very interested, not only in prediction marketing information, but lots of other kinds of information, too.

Agnes:

So then it seems like you've misidentified the target place for prediction markets, because you should be trying to help people do what they actually do want, not just what they say they want. So it seems like you just need to look somewhere else where people actually do want something that information would help them with. And then we could get a prediction market there.

Robin:

Well, there's a lot of theories to consider here, and that's part of what this prompts one to do, is to explore the space of possible theories of what's going on. But this is, again, my history is starting out with the naive perception that people should want information, and that we should look for people who want it, and then work on that. And then I've developed my... I think the story doesn't make sense.

Agnes:

story you just told is not coherent. Because if what happened was you thought, okay, what people really want is more information to make these crucial decisions. And you're like, here's a tool. And then they're like, now we hate your tool. And that's how you learn that they don't actually want that. And you're still pushing these prediction markets for these very things that you now know they don't want. It doesn't make sense. Unless y'all are new theory that they do actually want it even more secretly or something like that.

Robin:

So when people are hypocritical, it's often a complicated mix of what exactly they want and what they would be willing to choose. Often there is a thing they want, but they don't want to take it in the most obvious, direct way because that will get in the way of the image they're trying to project or the strategies they're trying to pursue. And so I would say that basically most organizations Management is mostly politicians who are part of coalitions, managing coalitions, and trying to manage perceptions about the things that their coalition has decided are its key agendas and priorities, and that that makes it harder for them to sort of just directly collect relevant information and do what's in the interest of the company. So in many, this is also true in politics, of course, the world is full of organizations where there's a simple collective interest of the organization or community, but a complicated arrangement of leaders and coalitions who are need to be, you know, convinced to support things where they have complicated agendas and alliances and strategies with respect to their, you know, what they're willing to support. That's the nature of our world of organizations. So I'd say, go ahead.

Arnold:

Well, this explains sort of the direction that this has taken, because that is with the introduction of prediction markets as a way to solve organizational information problems within organizations, like ways to find reliable predictions within an organization that's making a decision. Instead, we get these things like polymarket, where there's no organization.

Robin:

People just want to bet on- Exactly. We're just going to the sheep as the customer here, not the organization wants to know something.

Arnold:

Right. And so, of course, the polls are being structured around whatever people most want to bet on, not around whatever piece of information we actually need. And I take it that the reason for that is basically because in all the places where people could productively draw information from a prediction market, their status as elites hardly depends on the idea that they already have that information. or that they can generate that information themselves, whereas the prediction market would be tantamount to an admission that they are not, in fact, suitable elites.

Robin:

So a more familiar example of that is A-B testing or experiments. So there are now famously some companies like Google, et cetera, who do more A-B testing of different business options, marketing options, product options, things like that. But in fact, it's still really quite rare in the business world to do experiments. And the standard story, which I believe is that typically an organization for every question you have, there's somebody who's supposed to be the expert on that. You're supposed to find that person and ask them and they're supposed to tell you and you're done. If that person tells you, I don't know, let's do an experiment. They are basically admitting that they don't know as much as you thought they did. And you might wonder, well, why is this person in the job? Why don't we get the person who knows the answer to this, to put them there instead of this person who doesn't know. So in fact, even though there's an organizational interest in doing experiments, it can be hard to translate that organizational interest into actual experiments because individual people have an incentive to act confident and knowledgeable about things that in fact, experiments would usefully inform, but they'd have to admit they don't know.

Arnold:

But then, Agnes, the solution may well be to just go through the long slog of bringing prediction markets into the cultural fore to the point where an elite person in a company can say something like, I don't know, let's use a prediction market to find the answer. And that won't result in them being looked down on. as a fleet.

Robin:

So we have examples of that. So for example, focus groups are a standard practice in organizations where you do the focus group to get an answer to a question that somebody in the organization might feel inclined to have an opinion on, but they've learned in those organizations that you do not predict what the focus group will say because you don't know. And that for certain topics, you just always do the focus group and you don't pick some manager and say, what do you think? So they've learned that there are things that they don't know that they should try. And same for organizations to do experiments. They've learned that for those things, they should do the experiment and not just ask some manager. So organizations are capable of learning that there are certain kinds of information they should get from other sources besides asking whatever manager is closest to that topic.

Arnold:

Right. And so the antithesis of this would be something like the mythic Steve Jobs guy, right? Where you have Steve Jobs as this kind of like guru.

Robin:

Yeah. He's a kind of knows all, decides all, has the great intuition. Ask him anything and he just gives you the magical answer.

Arnold:

Right, right. And then on the other side is management structures in which setting up a system in which information is generated and no one gets credit for generating it might actually work. And then the question is, how do you get firms to start thinking of that as a good idea?

Agnes:

I just had a funny thought, which is that, like in the ancient world, there was the place you went if you wanted to ask a really important question, and it was the Oracle at Delphi. And, you know, there was like a priestess, and there was some gas, she was getting some noxious gases, and she gave you some answer, and you had to interpret it. And you, and we might think, um, oh, we're like really way advanced beyond that crazy thing. But in fact, what we do is like, we consult like the Oracle of our intuitions or something. We just have other Oracle. We're like, what do you think, Steve Jobs? Steve Jobs is kind of an Oracle. And the thing about an Oracle is you can be like, you can be like, well, it traces to a divine source. Like the divine source could be the gods, or it could be Steve Jobs, or it could be, I deeply feel it inside. I'm the divine source. And there's a way in which what the prediction market does is it rips that information away from any kind of claim at having that sort of a source.

Robin:

Well, actually, there was this book called The Wisdom of Crowds by James Surowiecki in the aughts, and he basically tried to say, well, the reason you should trust Marcus is the magical wisdom of crowds. and raise that as to a egalitarian virtue. He said, boo, centralized experts, yay, decentralized crowds.

Agnes:

You want the wisdom of the crowds. It's completely obvious, it's not divine, it's a crowd.

Robin:

Oh, no. Agnes, we're in a democracy. We're about to have a sacred event where we will decide the most important decision in the country by the wisdom of the crowd. That is us. No, we don't.

Agnes:

That's why we don't just go by the popular vote. We actually have a really complicated algorithm for figuring out who wins the presidential election. No wisdom. I was just reading Simone Bae today and she had this line, if the only way to heaven would be to join the party, I think she means the communist party, even though in her heart she's a communist, right, but she just won't join the party, or even the Catholics, she won't officially be coming out, if the only way to heaven is to join the group, I wouldn't go. She's just a deeply, deeply, deep hater.

Robin:

Well, that's an anti-authoritarian, anti-hierarchy structure, but that's the spirit of democracy and the wisdom of crowds. It's both energized by the hatred and distrust of authorities and hierarchies, and the virtue and sacredness of the crowd.

Agnes:

You're right. Hate and distrust authorities and hierarchies, and also hate the crowd.

Arnold:

But I want to connect this to something that we talked about at the beginning, which is that a poll, Nate Silver, and a prediction market represent three very different levels of information, right? The poll is data, Nate Silver is an interpreter of that data, and the prediction market is like a clearinghouse of interpreters. And so, in some sense, what we're talking about here isn't us choosing between two different parallel sources of information. When you recommend prediction markets to somebody as a way to produce information, you're recommending that they move from people like Nate Silver to clearing.

Robin:

An analogy might be Metacritic. Yeah. So many people get their movie recommendations from my favorite particular movie, Cricketic, and I do tend to trust Metacritic more because it's averaging over many different particular things. It's a forum that's neutral with respect to each particular method or approach, and it's just trying to aggregate the different individual opinions into some neutral institution you could just reliably trust without having to make decisions about who is how believable and expert, et cetera.

Arnold:

Right. So I mean, at least for me, okay, I really do have the sense that a lot of my own hiccups in sort of thinking through prediction markets and understanding them and understanding their value in a given context, which it's worth saying out loud that the value that these... Something like polymarket for the presidential election is not a prediction market in any sense that's useful and that you've been promulgated. But the value of these prediction markets really isn't that it's one source of information as opposed to others that are a lot like it. It's a way of taking those other sources of information and building an ad hoc something like an ad hoc academia, right? So if you wanted to know which direction your company should go, like for example, should you fire your CEO or keep him, the attraction of the prediction market isn't that, well, we can listen to the prediction market instead of listening to some managers. It's that we can turn our managers into a little mini ad hoc academia that will aggregate this information so we don't have to go through some process of vetting each individual manager's opinion and stuff like that, that we're leveling up in the kind of information that we're dealing with, informational source that we're dealing with, not just going for an alternative. Because when I go on Twitter, which I rarely do nowadays, and always with regrets. But when I do, what I find is people doing things like saying, should we listen to PolyMarket or should we listen to Nate Silver? And that's not a real choice. That's a choice between two very different kinds of information. And it's perfectly intelligible to just think they're both very good sources in the dimension in which they are sources, or they're both bad.

Robin:

Right. An analogous mechanism is an auction. Yeah. So you could, if you're trying to like get a supplier or a product, like do a lot of research and decide, you know, who do I trust and what looks better? But there are many cases where if you just hold an auction, you're basically saying, I don't know, I'm going to let this process decide for me. If the requirements I can specify are clear enough and then I do care about the price, I just say, okay, show me whoever's got the lowest price here. You meet my requirements. You're the winner. And but I am admitting that I don't know so much. So there is, in some sense, a pride hit or a status hit from choosing, say, to use Metacritic. So I think many people would say, look down on people like me who use Metacritic and say, you're just admitting that you don't have the taste. If you had good taste, if you thought you had good taste, you wouldn't.

Agnes:

Also, you have no friends. It's basically where do you get your movie recommendations from? Friends. So if you go to Metacritic.

Robin:

Well, my friends give me a recommendation. I look it up for Metacritic.

Agnes:

Right, you don't have any real friends.

Robin:

Okay.

Arnold:

I mean, that's true of insurance companies too, right? That is, we've just gotten used to the idea that to buy an insurance policy is to admit that in a crisis no one would support you.

Agnes:

Yeah, yeah, right.

Arnold:

And the police, right? To support the institution of police force is to admit that if you were attacked, no one would come to help you unless they were paid to do so.

Robin:

But prices like in auctions often serve this, like you put up a rental price for the apartment and you take whoever is willing to pay the rent. And you don't just put up a thing and say, come talk to me and we'll negotiate. You put up a price. And so prices are often ways that you admit that you can't really tell who's best about things. You're going to let the price decide.

Agnes:

Okay, intentional question. I know that you decide what movies to watch and what TV shows to watch based on Metacritic, but I'm struck that you don't- That's my friend's recommendation. I combine them, remember. Okay. Well, wait a minute. Okay, now I'm going to go back to the original question because that's what I want to know is how you combine these things. But for the moment, my sense is that you don't decide what books to read that way.

Robin:

Well, there isn't a good Metacritic for books. If there were, I might well use it more, yes.

Agnes:

Is there like Goodreads with stars?

Robin:

Yeah, it's not the same. I mean, again, Goodreads is going on just ordinary person's value. There's the other, IMDB just has user ratings, and that's different than Metacritic. Metacritic is averaging the professional reviewers. And I am gonna trust them.

Agnes:

If you were a Metacritic for books, you might use it.

Arnold:

I might, well, yes. Both Metacritic and Rotten Tomatoes have both reviewer ratings and user ratings, and that's useful. Like often, if you- No, both. Very, very high reviewer ratings, but quite low just fewer ratings, then something funny's going on. It's like... Well, it means you will respect it, but it won't be much fun.

Robin:

Yeah, exactly. Right. Right.

Agnes:

Okay, but so I want us to go back. We only have like one or two minutes left, but... I want to go back. So you have friends and you have Metacritic and you somehow take them both into account when you choose a movie. And so now I want you to talk me through. I look at the polls. I mean, I don't do any of this. Arnold, he tells me, looks at the polls and he looks at Polly Market and he's like, how do I, you know, how do I fit these things together?

Arnold:

No, no, no longer. I no longer think they can be.

Agnes:

Right. Well, no, but Robin thinks they can because he does it with the Friends and the Metacritic. So now tell us how to do it with the polls and the prediction markets.

Robin:

So for a movie, it's an intrinsically multidimensional choice in that I have many factors I care about for whether I like a movie. It's artistic quality and reputation is only one. It's a big one, but it is only one. If I want to know who's going to win the election, that's really one-dimensional. There isn't a whole bunch of related things I'm trying to figure out about that. It's just that one thing. So the more clearly one-dimensional it is, the more I'm willing to just go with the one-dimensional choice about what to do. But I think I might reframe it this way. I would say for the vast majority of topics in the world, you should not be a specialist. So you should just take a consensus on things that you're not going to specialize in. And then for the limited number of things you're going to specialize in, then you need to look at more than just a consensus because you're going to contribute to consensus. I would say that's the division of labor. So we each decide, what are we focused on? And then we will do more research on that and then offer subtler contributions. But other people who are not focused on, they just need to look at an aggregation. And so we should be contributing to some aggregation. and influencing it because that's our contribution in the world, but people who aren't focused on this should just accept. So like in academia, if I publish papers in IR, I may well disagree with other people who publish papers in that area, but it's fine if there's just a general perception of the consensus in that area roughly that other people get out of the net effect of all of our publications because they're not specializing in it. So I might say to Arnold, look, Is this really something you want to specialize in? Just take the aggregate estimate and don't mess with this topic because you're not likely to be someone who's going to be contributing to this topic. You're diving into more details than you need to here. What's the point?

Arnold:

The lesson here is looking at polls and maybe even reading analyses of polls is that's wolf behavior and I'm not a wolf. Right. If I were to contribute to the market, it wouldn't be in a wolfy way, that's why I

Robin:

Right, but apparently a lot of sheep think they're wolves, and that's what motivates a lot of this research into topics by more extraordinary people, right? They're not just taking some consensus and walking away with it. They're trying to do their own little research to it as if they were kind of wolves.

Agnes:

Okay, very last question. Say you think you're a wolf. How could you find out if you're really just a sheep?

Robin:

Well, if you have available a betting market, it's a great tool because you make actual trades and you see if you win or lose. You just win, of course. As an academic, potential academic wolf, you try to write a picture and you send it out for review and you see whether people think it's wonderful or crap, right? How do you behave?

Arnold:

Yeah, so in the rationalist community, it's a common practice to bet on as many things as you can to see how good of a wolf you are in general, right?

Robin:

And there's a lot of status to be had from being... Actually, more precisely, you have some opinion about how much of a wolf you are, and you might think you're only a tiny bit of a wolf, but then you should still bet to see whether you're right. Are you even a tiny bit, right? The whole point is you should calibrate your level of quality all the time on all sorts of things, and the betting will do that. And then if you're tempted to try to become more of a wolf, you should track yourself as you think you're rising in quality there by betting, and not just betting, but of course, writing and circulating among critics, et cetera, all the sorts of ways in which you could track, well, how much do I know here compared to other people?

Agnes:

In a way, the really important thing is not whether you're a wolf or a sheep, but knowing how much of a wolf you are.

Robin:

Exactly. The biggest problem is all these sheep who think they're wolves. They're really losing.

Agnes:

I mean, there would also be a problem if a wolf thought that they were a sheep, and then all this money they could be making.

Robin:

Absolutely. That's true. Alternation they could be delivering. Yeah. Exactly. That's maybe the polymath intuition of try lots of things out, see what you're good at.

Agnes:

We should stop. We are over time.

Robin:

All right. Thanks for talking, both of you. Yeah. Thanks, Robin.