Today we're talking about a reading that I picked called Inadequate
Equilibria. It's a whole book, which is longer than the reading you picked so
I apologize for its length. And I have to disclaim that it's written by an
associate of mine. He was a co-blogger with me for many years, and someone I'm
fond of. But nevertheless, we both read it. And now, I should ask you for your
immediate summary and take.
Yeah, I had a one sentence summary I wanted to give, which is, our systems are
broken so I'm allowed to trust my reason. That's the argument of the book as I
see it, right? So, there is some kind of prima facie problem with reasoning
your own way to problems, like trying to think through problems and come up
with theories. There are some thought that you're not allowed to do that
unless you're living in a broken world. And then– so then the first part of
the book is like you are living in a broken world. And then here are the ways
that it's broken. And the second part of it is, that's why it's OK to trust
your reason. That’s more than…
So, I wrote a blog review of the book years ago. And one of my main points was
just it's basically two different books, which aren't very connected. The
first– but those two little clauses of your sentence. The first book is the
book of the title, where and how civilizations get stuck? And the second part
is, why then that you should be allowed to trust your reason, or to disagree
with other people, why you're justified and disagreeing with others. And both
of those parts of the book have a lot of discussion, but there's not actually
much connection between the two. Because you’re merely saying why.
OK. Let me say the connection.
So, if you were living in a society that had, like, optimized all of its forms
of coordination, so that everybody on every purpose was like coordinating as
well as possible, that would include informational coordination. And so, in
that world, you wouldn't be allowed to think for yourself, because instead of
trying to think through something, what you should think is, the total product
of your society thinking through that thing would be better than yours.
Well, so close, I'd say, that is you would inherit a consensus on a wide range
of topics that would be pretty good. And then you would be able to make minor
changes, but you would be able to make. So, he discusses financial markets as
a prime example of a place that you should trust. And that means you shouldn't
think the financial market prices are wildly wrong, or that you can easily
double your money in a year via your clever insight, right?
Nevertheless, the way that entire world functions to improve is that
individuals contribute to it. That is, individual traders do find a little bit
of information and make a little bit of trade and making minor adjustment to
it. So, each of them needs to believe that, in fact, they are making a small
improvement on average. And the net effect is improvement, collecting all
these small contributions. But you just couldn't believe you, you had a– they
had a huge mistake and that you were– it was very wrong and you had a big
I think actually with– I think that's right. But I think the thing he's even,
he's more interested in is not– it's both can you make a large correction, but
also, can you kind of make corrections all over the place? Like, you have to
Right. So their financial people will specialize, like, they will find one
very particular area, and I try to improve that and leave everything else to
everyone else. So, you're right, that if we were adequate, then you'd only be
able to make narrow, small improvements. But conversely, it's not true that if
the world is broken, that you need to disagree with people. That is the world
could be broken and everybody could just know that. And when you claim that
you had a new insight or a deviation, everybody would say, “Well, of course
you can do that. Of course, that's possible because the world is broken.” So,
but that's not the world we live in.
So that's to me, the– to me, the interesting connection is that in addition to
the world being broken, we believe it's not, or we think it's not, or we say
it's not. And that's the lever that we would force him to then in order to
take advantage of a broken world or to improve on a broken world, at least for
himself, he needs to reject that large shared consensus.
So, the shared consensus is that we believe that it isn't broken?
That isn’t very broken. Yes.
So, I think that's impo– so…
So, we could take one of his favorite– his favorite example of the book is
when his wife was having trouble with dark. And, they try the simple fixes to
winter being dark and made modest light improvements. And then they moved to a
light continent for a while. And that helped, but the small lights in their
apartment didn't help and so we thought, “Well, let's just make a lot more
lights in the apartment.”
And that worked. But he says the literature doesn't seem to represent that as
a possibility and you might say, “Well, there's all these doctors who
specialize in this problem. They don't talk about it. What makes you think you
could come up with some improvement here where they haven't?” So, that would
be this widespread perception that he couldn't come up with such an
improvement, because they're presuming that the world of doctors recommending
how to deal with dark problems is not sufficiently broken, right? It's the
believing that the world is broken, that then emboldens him to try this
radical fix. Whereas most people wouldn't do that, because they wouldn't
Sure, I think that's right. But that establishes the connection between the
two parts of the book. That is…
Right. So that's what I'm saying what I think the strongest connection is, is
that we’re living in the world where people…
OK, so right. We both…
Where people don't believe it's broken. They don’t talk as if it is broken.
Right, right. So, it's both that the world is broken and that because people
don't believe that everyone is not going around trying to fix it.
Right. They're not as aggressive or eager or willing to try all these things.
So that's, to me, the key connection between the two sides of the book,
although he didn't play this out explicitly. And so, I think that helps us
think about the second part of the book so this where we had to focus on that
is when are you justifying and disagreeing? It’s when…
Sorry, can I just ask about that?
Suppose that people did generous– lots of people read his book, right? And
lots and lots of people are convinced. And so, they all do their own, like,
you know, he has a kind of like, do-it-yourself ethos, like, do-it-yourself
medicine, do-it-yourself dieting, do-it-yourself. Right?
But suppose the people adopted this do-it-yourself ethos, it still could be
the case that it wouldn't be so easy to sort of like aggregate all the
information that they got from their do-it-yourself trials, so that…
This would remain broken. And it would remain true that it was reasonable for
people to trust their reasons. So that shows that your premise, which is that
most people don't believe that it's broken isn't actually doing work.
So, there's– the difference between sort of being willing to explore something
and disagreeing with other people that that. So, you could say, you know, as
soon as we heard that he had this idea for lights, we could have all said,
“Yeah, why don't you try that? That sounds like a good idea.” And then we
wouldn't be disagreeing with him. But we would be still supporting his attempt
to reason for himself. So, thinking for yourself isn't the same as
disagreeing. So that's the key distinction to make here.
When do you think you count as disagreeing with someone?
Well, so I've had a bunch of papers that I've published on the topic of the
rationality of disagreement. And so, there's this large literature, and a lot
of which has been done by philosophers. And so, one of my papers’ title is You
Can't Foresee to Disagree, or the title of a blog post at least summarizing
that. So, people ask, can you agree to disagree? And there's a famous
economics paper from long ago, a Nobel Prize winner on that.
And so, agreeing to disagree would be some situation where we're all fully
aware that we disagree and nevertheless we disagree. And my variation is, you
can't foresee to disagree in the sense that if you're about to have a new
opinion, I don't know what it is, I can believe that it will be different than
mine, but I can't predict in which direction it will be different. So, I can
completely accept that you will go learn something and that the thing you
learn will change your mind, and therefore you will disagree in the future
with what I believe now.
So, it's not– I don't have trouble with expecting that difference or
disagreement, but I can't predict it. I can't foresee it. So then, in this
context, you might tell Eliezer, to go ahead and think about it and come up
with something. And you might think that whatever he came up with was
reasonable. And it would differ from what you think right now and there'll be
that disagreement, but that wouldn't be a problem, that wouldn't be agreeing
That didn’t really answer my question, which is, when do people– what I mean
is, what does it take for two people to count as disagreeing? So, suppose
someone on one content believes P.
And someone over on the other part of the world believes not P, right? And
they're never going to meet. In fact, the one guy has never asserted P and the
other guy has never asserted not P. It’s just that they would be disposed to
assert those things under the relevant circumstances. You might think that's
not really disagreement.
Right. You're right, it's not disagreement. So that's the key point about
agreeing to disagree is the puzzle not to disagree. So merely having two
estimates isn't that puzzle, it's being fully aware that you have the two
different estimates. That's the puzzle.
Let’s start with this. It's possible for people to disagree. Do you agree with
me about that?
Of course, of course.
OK. So sometimes people actually, in fact, disagree. It's not just when
they're on two different continents, but presumably, when they're engaged in
some kind of shared intellectual enterprise, like they're trying to figure
something out and they don't think the same thing about it and they didn’t
know that the other thing is the different things.
The word disagreement is a bit ambiguous, which is why we're using these more
precise versions. So, I would take disagree, in a broad sense to just mean,
you have different estimates, even if you're not even aware of them. So, but
you might just say, well, that's not a disagreement, that's just a difference
of opinion, fine. Difference of opinions exists when two people have different
opinions. And then you might say they disagree in a context where there's some
sort of conversation between them about it. And then we could– that's still
not quite strong enough for these theoretical results to apply. And so, the
more precise thing we talk about is common knowledge of difference of opinion.
OK, but I guess the thing I'm interested in is disagreement, not difference of
Or common knowledge of it.
I am interested in common knowledge of it. And I think– because that is I
don't think it counts, really as a disagreement unless there is common
knowledge of it. And I think that that's – it seems to me that's both possible
and rational to disagree in that way. And not only is it– not only is it
rational for people to disagree, it's rational for people, it seems to me
because I do it all the time and I think I'm rational at doing it.
You intentionally disagree that is to deliberately adopt the opposing view,
simply because it's the view that is opposed to the person who you're talking
to. And there’s something that– there's something that Eliezer talks about at
the end of the paper where he talks about this danger of not wanting to rush
the process of agreement.
You might think like the actual real-world problem is that people agree much
too easily with one another. And that the rational thing to do is to
intentionally disagree that is to adopt the opposing position so as to inquire
into the truth.
So, I'm going to take the philosopher stance here, in this sort of
conversation, which is to say, well, let's pause and make sure we're clear
about our terminology.
Because that is the issue. So again, there's a difference of opinion, where
people have different– but we even want to distinguish opinion. So, one kind
of opinion would be the thing you would say to prompt and further a
discussion. The thing you might say that get would people to react that would
get people to respond, the thing that you might defend. That's one kind of
opinion. And then another kind of opinion is the thing you would act on when
substantial stakes were in play, right?
And so, this– about difference of opinion in this book is primarily about that
second concept about the actual beliefs you would have to take substantial
action. And it's not primarily about what you might say to further a
discussion. So, the puzzle, I mean, it's completely understandable why, in
order to make a discussion go forward, you might take different positions and
explore their conflict. But you don't necessarily have to be inclined to take
different actions outside the conversation, because you took that conversation
OK. I mean, I think that in a really honest disagreement of the kind that I'm
describing, you would be committing yourself to that position until it gets
refuted, right, so. And you would take action on it if it never got refuted,
like that's what it is. You're exploring this idea, and you're like, you're
going to stake your claim in there until it's proven wrong, which presumably,
if you can think of arguments against it, it easily will be, but still.
But in any case, it seemed– the reason I brought up this whole thing of
disagreement is that you were talking about how, you know, the question– I
said, even if everybody believed the first part of the book, the second– it
would still be rational to trust your own reason, because we wouldn't
necessarily be able to aggregate all the information that people got from
their individual trials. And then you were saying, “Well, the issue was about
disagreement. And whether or not you disagree agreed with what most people
thought.” And that's why I thought, “OK, well now we're talking about arguing
rather than mere differences of opinion.”
So, my actual favorite example of the broken world is nothing that Eliezer
mentioned in his book but it's really easy to grasp, which is on the freeway,
when you're driving past a city, there are often signs which say which exit to
get off at to reach a particular destinations, like a stadium or a convention
center or something like that. And those signs are often just not the fastest
way to get there. There are some signs are official advice, right up there on
the road, but your GPS, or your map will quite often tell you another route is
Now, if we all know this fact, then if I'm sitting next to you while you're
driving, and you don't follow the advice on the sign, I don't have a criticism
for you. I don't disagree with you because I figure well, of course, those
signs are wrong, often, and you are taking a different exit, because you have
another reason. And you are in a sense, disagreeing or being in odds with this
advice that civilization is giving you on the sign.
But if we all know that advice is not very reliable, then among ourselves, we
won't be disagreeing about the actions you take defying the sign. So that that
would be my concrete example of how in order where we all accepted that the
world is broken and the signs are wrong, then we will, you know, be fine with
somebody else defying and ignoring the sign. So, there's a lack of
disagreement there. And then you’re thinking for yourself.
So, we're talking here about if you’re being within my sense, that is verbal
argumentation, that's what's important there. Because I think you keep going
back and forth on whether that…
Well, I mean both, but I guess I sitting next to you won't disagree with your
choice to pass– I won't have a difference of opinion regarding your choice to
pass the exit that had the sign saying we should get off there.
I see. So, your thought is in the world where we all read the first part of
Eliezer’s book and we're convinced by it, there would be a set of verbal
disagreements that didn't happen. Whereas in the world where we don't, we have
those disagreements if somebody – so the person who trust their own reason, in
the first– in the world where we haven't read the book, they have to have a
bunch of arguments that the other person doesn't. That seems to be a trivial
difference and insignificant difference between the two. Like so is it just
like, well, just like having those verbal arguments.
Well, I mean, arguments resulting from their sincere– I mean that their
sincere beliefs differ, and then they argue as a result, so I'm equating those
two in this context. I'm saying, in one world, we think the signs are right.
We assume that usually, and if so, somebody defies the sign, they're likely to
get pushback from other people around them, saying, how do you think you know
better than the sign?
They're having verbal disputes.
Driven by differences of opinion.
Not just an attempt to discuss it, right?
It could well be differences of opinion in the world where none of us believe
the sign. We could have different ideas about how to get there, right? It's
just the we don't know about them. We don't have the verbal disputes about
Or, they’re a different character of disagreements. So again, you know, he's
focusing on this kind of disagreement where there is some sort of authority,
and the authorities have a certain stance, and that you're supposed to go
along with the authorities, because the authorities know better. So, a great
many of his examples are of that form, you know, the doctors recommended one
thing or the bankers’ monetary policy people set another, etc. And so, I mean,
he often – and in the latter part of his book, he talks about it as sort of a
status, you know, policing issue, that we often frame these things in terms
of, who do you think you are to put yourself up against these authorities?
Yeah, I mean, to me, this is just – it's very odd that it would come down to
this like, some kind of allergy to verbal disputes or something. It’s like
what it comes down to, like, what if you like having verbal disputes? Like
what if that's a plus for you, then you would be more likely to do the
do-it-yourself lifestyle in the world where nobody read the first part of the
I'm not sure we're disagreeing, so I guess we should just make sure we go
methodically through the possible positions here. You know, so we agree, I
think on what it means for world to be broken.
I actually don't know that we agree about that. We haven't really discussed
part of the…
We rough… no, we've been presuming that, I guess, that a broken world is one
where the sort of messages you will get from the usual authorities are not
very trustworthy in the sense that if you have substantial stakes and are
willing to spend some time thinking about it, there's a substantial chance
that you will be able to come up with something better.
But can I ask about that? Because I don't – we probably don't agree about it.
Because I don't agree with the first part of the book.
So… or I've just been, I've just been holding on the argument [0:20:31]
Right, right. Well, no, no, that's fine. Yeah. OK.
So, I thought that was your big problem. So, I thought I would it solve that
for you, the connection between the two. But… so, it seems to me that what he
notes is that, like, with his wife, you know, when she had this seasonal
affective disorder, he was able to improve on the medical establishment. But
he doesn't tell you like all the cases, when he wasn't able to improve, like,
I bet he's been to the doctor a lot of times, and I bet a lot of times doctors
said the right thing. In fact, he even gives an example of that.
When he thought he saw– his girlfriend saw these spots, and he thought he
could diagnose them in the doctor, and he didn't. So, it's like, I think that
what he shows you is that our system has like systematic failures, like it's
going to fail, in many cases, but it's going to succeed in many, many, many
cases. And maybe it's just succeeding in many more cases and so like, it's a
teeny– teeny tiny little bit broken.
Why not think it's– just like with the Bank of Japan, like, OK, that was
crazy. And that was like, it was like a big mistake. But like, think about all
the banks of all the countries in all the world's most and think about even
Japan at other times, right? So, it failed for a little while, but like,
mostly, most of the time, most of the system succeeds, it just that there are
these little bits, and you can focus on those and then say that system is
So, he's definitely not claiming that, you know, you should just typically
disagree with what the authority said. Right? He's just– he’s certainly not
recommending the contrary of position and just believed the opposite of
whatever they said, that he thinks that would go very wrong.
Or not– he's not even saying believe what they say 50% of the time. That is,
he doesn't think that they're worse than… that is there’s only as good as
So, I mean, he thinks that being wrong 20% of the time is a really big
Does he think– do you think he thinks that medical establishment, in terms of
what diseases it diagnoses you with is wrong 20% of the time? Because he
doesn't give evidence for that.
But so, he doesn't give evidence for overall rates at all. Although, I mean, I
think we do have evidence for overall rates. But he doesn’t really have that.
OK, but we're looking at the argument of the book. And what I'm saying is, it
seems to me a substantial flaw of the first part of the book that he doesn't
give evidence for overall rates.
So, he gives a lot of compelling examples.
And in his dialogue, he sort of goes through some pretty big examples. But of
course, I'm not sure how much it matters for his thesis, the exact percentage.
I mean, what he's trying to address someone in the position of himself there
with his wife, with she has this problem and he wonders, how much might it be
worth to go think this through and come up with my own alternative? That's
what– that's the person he's trying to address. And…
Yeah, but you don't know that you're– when you're that person, you don't know
you're in that case, right? So, look, imagine my kids, somebody interviews my
kids, and they're like, “Tell us all the worst things that your parents did.
Tell us all the like bad examples of their parenting.”
And my kids list all our terrible examples of parenting. And my kids are like,
this system is broken, right? My parents just make bad decisions and I got to
do this parenting thing myself. I got to do my own upbringing, because clearly
my parents, I'm living in a broken system. That would not be a good argument.
But he's not making the analogist argument that you throw away the world you
live in and go live in the woods and try to reconstruct your own civilization,
right? He is– he does recommend, I think that you continue living in this
civilization, which means that you continue to accept most everything in it.
But nevertheless, the question is, if you have a particularly large problem in
front of you, how willing should you allow yourself to be to consider that the
usual authorities answer might be wrong. But he goes into a lot of discussion
about how you should just definitely not, you know, authorize yourself to
figure that whatever you think of is probably better.
He talks a lot about that you need to test yourself that is you need to find a
hypothesis for something about why the authority’s advice is wrong, and then
find a way to test it quickly. And, you know, throw it away when it fails the
test. But if it passes the test, then you've got to win. So that, you know,
it's not about the initial full confidence that you have an answer. It's more
about being willing to just try things out.
But I mean, I think he thinks like if his view is “Look, when you are dealing
with a major calamity in your life, you should be like kind of holding the
world to much higher epistemic standards and be less inclined to trust and
must be more inclined to do your own research.” That's common sense and
everybody already does it. That is these parents that he's describing with the
babies that have, you know, don't get that like special fluid, like, clearly
plenty of them actually did their own research, even without reading Eliezer’s
book, right? And they did it exactly – and they did it for the same reason
that he did it in the seasonal affective disorder.
But apparently, most of those babies’ parents didn't do that and they died, so
that is a minority of those parents.
Right. But you might think, look, the issue is that when you're in one of
these cases, like the seasonal affective disorder case, you don't know that
you're in that case, right? And so, you could well be in the case where you
will drive yourself insane, trying to find your own solution to the problem.
So, one thing you have to do is make an assessment of which sort of a case
you're in. And I'm sure the parents whose babies died did put effort into it,
right? They maybe didn't put as much effort as those other parents because
it's the hard judgment call. And I guess I think I don't see how reading this
book would tell one of those parents how much effort to put in.
Oh, he's making a judgment that I roughly agree with him, that when he's
talked to a lot of people in the world over a lot of years, as I have, that
when he brings up a suggestion for something that goes against the authority’s
advice, people at that moment, bring up this authority deference issue.
That makes sense, because they're not in a calamity at that moment. And his
advice is only take the serious– so far is only take this seriously if you're
in a calamity. So they’re given the right…
I don’t think calamity is the right word but it's something important enough
to you that it's worth thinking about.
Right. But it's not important to the people he's talking to probably.
No. So, I think in many of these contexts, it was important to the people he
was talking to. So, he talked about someone who had a startup, for example,
he's talked to several people that had startups and their strategy and his
advice that they should try something out and test it. I would– I mean, he
didn't mention it, but he is like me a cryonics customer so he would share
that contrarian position on that topic, which to, in some sense is important
to everyone, or plausibly important to everyone. And his main research area in
his life was this AI risk area, which he thought was very important, but
important to everyone. And that people sort of shut him down initially by
saying the authorities would have been on– or if there was really a problem
here, the authorities would have been all over it so you must be wrong. So…
I get that it's once again, like, as I say, this kind of rationality base
rates thing is more of your thing than mine. But it still seems to me I'd want
to know, well, how often are people overconfident as opposed to
underconfident? I grant he can find me cases, maybe many, many cases in which
people are underconfident, right? But I bet you can also find cases where
people are overconfident. And then like, you know, is the idea here like,
well, he's going to give you meta-advice about how to navigate that in the
abstract, whether to be overconfident or underconfident. Like I doubt it,
like, I think it's going to be a matter of the specific case.
So, to support your position, that is, if we accept the description of this
book, as these two books, the first on the world is broken and the second is,
therefore you can agree to disagree. I wrote a book review, as did Scott
Alexander, as did Scott Aaronson, and all three of us said, well, that first
part of the book is much better than the second. That is, the world is, in
fact broken often, and that's an important thing to understand. But they– all
three of us took more issue with the second, connected conclusion that
therefore you are more authorized to disagree with people.
So, Scott, for example, discussed how a relative of his, you know, was, or I
mean one of the Scotts actually, you know, was tempted to buy timeshares and
that they would go listen to a timeshare pitch. And, they would seem, it
sounded like– sounded good. And then people would say, like, “Yeah, but look
at all the past times when people were fooled by this stuff and have lost
consistent money and, it looks like a pretty bad track record. Maybe you
shouldn't trust your judgment here in this case.” And, you know, they were
saying, “Trust this wider judgment.” And there are lots of– that's basically
the situation here is that most of the reviewers, these three reviewers at
least agree that the world is in fact broken and that's a strong message, but
how much that authorizes you to defy it in any one case is much less clear.
I mean, I guess, as I would prefer to rephrase the first part, the world has
some imperfections. Right? That's all he's argued for. But I am sorry, there's
another thing, the imperfections are somewhat stable. Right? So, there’s…
Systematic, they’re system– and they have understandable causes.
But I meant to say stable, which is to say, that they are not going to get
easily self-corrected, which is not the same thing as saying they're not going
to get self-corrected, because every single one of his examples is one way
either did get corrected, or I predict will get corrected, and like, people
are going to figure this thing out about seasonal affective. Like, either it
was just a fluke case and his which is totally possible, right? That this
slight thing didn't actually care her or whatever. Or like, it'll eventually
get figured out, and then we'll eventually have those special light bulbs,
maybe we even have them now. Or the thing with the babies, like, eventually, I
think just like the Bank of Japan eventually figured it out, it will just like
take longer. But nonetheless, if these inefficiencies are in the system, they
could stick around for quite a while before they get corrected. That would be
what I think he's sort of shown.
So, the world is imperfect, and the imperfections are somewhat sticky. That's
what the first part of the book says. How prevalent the imperfections are,
like relative to all the ways in which it works. There's nothing about that,
Just like in the second part, all he says is some people are underconfident,
not, how prevalent, like how much underconfidence is there relative to how
much overconfidence is there. So, to me, neither of those two theses as such
are that strong. The thing that's most interesting about it to me are the
specific mechanisms of the stickiness of the imperfections, right?
Right. I think what you see those mechanisms, you see the possibility that
there's a lot more problems that he mentions, right? You see the possibility
that this is a much larger problem than these mere examples
Sure, but like, I don't feel like the basic inference in the book, like here's
another way and here's a kind of psychoanalytic way to read it, there’s
something like, it's like, there's like a little boy who wants to take his
toys and go home. And he wants to say, “Yeah, that I have a good reason for
doing that because this place sucks.” Right? So, it's like…
You might think, look, what you're supposed to do is take these systems, which
mostly are awesome, and produce great results, and try to work within them to
improve them a little bit. Right? So like, you and I, we're stuck in the giant
system of academia that has like tons and tons of flaws, right? And maybe one
day it will collapse, because of all of its flaws. But until that day, we're
in it. Right?
What do we do? Well, we give students the best advice we can, we teach as well
as we can. We… like we don't just like opt out, right? And in fact, you might
think a big part of why people want status, right, within a system, why
there's such a quest for status is that once you have status, you do have some
ability to change a system.
Now, it's also true that the quest for status turns you into the kind of
person who's not going to change the system. So that's a big problem, right?
But it's possible to fight it somewhat. And I'm sure you see, as I see around
me every day, people who are trying to fight it and improve it, right? And
look, if somebody doesn't want to do that, if he doesn't want to improve this
because he thinks it's too broken, even though he has given us statistics
about you know, how broken it is relative to the whole, that's fine. I think
you're just allowed to take your toys and go home and think for yourself and
come up with theories.
And like, if you don't want to have a disagreement with me about that, that's
OK with me. But it's some– there are some weird thing where it's like, yeah, I
want it– I don't want to work within this broken system. I want to think for
myself. And also, like, I somehow, I'm worried you're going to criticize me
for that. So, I want to produce this justification, that I should be immune to
criticism or something for that. It's like, there's too much self-justifying.
One of the reasons that I suggested this reading for us is that we have
previously discussed my many attempts to diagnose and prognose various big
And so, this author clearly believes that there are big problems that could be
fixed. That is that there's a lot going wrong. And so, if we had better
institutions and mechanisms, we could get a big gain from that. So, he agrees
with me in that sense, but he doesn't go numerically, you know, as far as you
or I would like to sort of argue that things are substantially broken.
And so, I would quantify as saying, yes, things are slowly getting better and
eventually maybe most of the problems will get fixed, but the rate at which
things are getting better is plausibly a factor of 100 less than it could be.
If we had better institutions, we would be getting better 100 times faster.
And the opportunity cost of that is enormous. That is, you know, had we been
getting 100 times faster from, say, a thousand years ago, we would be a lot
further along now. Or even from 10 years ago, right? If we had done a thousand
years’ worth of progress in the last 10 years, then we would have solved so
many more of these things.
So now, you know, I haven't proven that to you, but I could give some
plausibility arguments. But fundamentally, if that's the level at which things
are broken, then there's enormous potential payoff in figuring out how to fix
that. And looking at the larger structure.
That might be true. I agree that it's useful to bring that into connection
with this book, because in a way, right, if you take, like, his examples, the
parents and the… we should probably say that with this example, because some
people might not read the book. There are some babies who are born with some
rare disease, where if they get this special kind of formula, they're fine.
But if they don't get that kind, they die. And that special kind of formula is
like not FDA-approved.
So… but those parents, some of the parents, right, did tons of research,
probably all of them did some research, and most of them did some kind of
research. And some of them did enough research to learn this fact, and go and
drive to Philadelphia in the one place where you can get this special formula.
Anyway, so those people are really motivated, right? They're motivated to buck
the system. And he was motivated when his wife had seasonal affective disorder
to like, you know, reinvent the wheel, so to speak. But most people, most of
the time are not motivated to do that. And it's rational for them not to be,
that is they're taking the correct approach, at least as far as like, as far
as he's argued, right?
And so now, suppose you are like, look, I have this recipe for how we can burn
this system down and have a new system, right? And then you'd bring this to
someone, right? And then the question is, what's the rational response of that
person? And suppose they find your arguments convincing, I usually find your
audience pretty convincing.
*OK, problem number one, I don't feel motivated. That is, I feel like I don't
care. And I'm not saying– I'm not even saying that’s rational judgment. I'm
just telling, I'm just reporting that phenomenology. And I think part of it is
that it doesn't really seem to me that there's anything that I can do about
this. And it doesn't seem like it's a problem for me. And you know, it may be
a problem for everyone together.
And then if you were in some way addressing me as part of everyone together
that might move me. But addressing me as an individual, it sort of seems more
like what you're trying to do is get me on your team of people who
ineffectually think that this would be a better way to do things. And I sort
of don't see the point of that.
So, in the chapters where he has the three characters doing their dialogue.
You know, he has a spokesman for what he calls a cynical economist explaining
to an alien, all the processes that are causing these things to go wrong. And
the alien is presented as indignant or incredulous that things could possibly
be this bad. One of the mechanisms he talks about, which is easy to get your
head around is the Overton window. He says, “Well, there's just a certain
window of policies you're allowed to talk seriously about as a serious person
in public.” And say, “The New York Times decides what those are based on their
reported judgement about what you would be laughed off the stage, if you said.
And that that system sort of limits what kind of things we can consider.
And he says, meta discussion that is a discussion about how the system itself
should change is outside the window. That's beyond the pale for serious policy
discussions. And he gives examples of how when something was finally let into
the Overton window, it might take over very quickly, because a lot of people
thought it was plausible, they just weren't allowed to talk about it. And he
has hopes that say drug decriminalization will come within that window,
And so that's a– not only a concrete example of an institution that's
problematic. It's an example of exactly what you're talking about, where
you're saying, well, if other people were talking about this, then I may be
motivated to join them. But if you just want to talk to me privately, and it's
not in the New York Times that I want to hear about it. I mean, that's the
Overton Window, you're applying the Overton window to your own motivations.
But let me explain why not. So, you've argued with me about cryonics. And I
would say you've shifted me over, like, I'm at least 10% more likely to do it
than I was before talking to you about it. And the reason is because that's
something where I could just choose to do it. Or, I can just choose to pay
this money, and then my brain will be frozen. I don't view that as like
outside the pale of consideration. Like, if you were like– and I sort of think
that if you wanted to propose to me anything that I could do and give me like
a good reason why I should do it. I think I'd be open to considering it.
But instead, what you want to do is have me consider something that where I
say to myself, as you're beginning to try to convince me, I say to myself,
“Suppose that Robin were to fully convinced me of this, and I were to become
100% persuaded of the truth of this claim, what would change?” And then I
answered myself, “No.” Then I'm like, “OK, I don't care.” Right. So that's how
I feel about these things. So, what I'm saying is because of because of the
Overton window, as you say, I don't think that my becoming convinced of this
actually matters. Whereas my becoming convinced that my doing, getting my
brain frozen would be a good idea, that would matter because I could do it.
So, one thing is that some institutions we might consider would be ones that
would break the Overton window obstacle, that is, we might be able to overcome
this Overton window obstacle by some of the institutions…
You mean, once we instituted the institution.
Not beforehand when we would need that in order to institute it.
Well, so I mean, think of Eliezer’s things with the lights, right?
If at that moment, he hadn't thought of the idea that he could just do it
Then he might have had your reaction to say, what can I do? And so, the key
idea is that many people are motivated to like, figure out how they can
improve the world and they are wondering what to do. And for a lot of these
ways that we might be able to improve the world, the limiting factor is
actually just doing small scale trials, like this small-scale trial. And that
is the sort of thing people can do. But they have to do it in part out of some
altruistic motivation that they're trying to contribute to innovation that
will then spread farther.
So, I mean, you could participate in small scale trials of things that if they
worked out in small scale trials could spread farther, and then make a bigger
difference. And that would be– that's what – that's major– Eliezer’s main
recommendation as well is consider yourself able to disagree enough to come up
with something you could test. And he really emphasizes that, you know, find
things you can test and test it often and even test yourself by betting,
right? And he says, you know, just think of things when you think of
something, try to make a bet on it, because that will really make it make you
remember when you were wrong and when you were right.
Yeah, so like, I mean, there are now to sounds like you're sort of giving me
advice for like living in a certain way. And at this point, I'd be like, well,
let me look at the people who have taken Eliezer’s advice, right, including
Eliezer. Right? And let me look at how they live and like what giant
contributions they've made to humankind and see if that looks like plausible,
like, that sounds like a good– and I don't actually– I don't know about him,
But he does say like, you know, he says something towards the end of the book.
Like, I want to know how far has this modesty approach gotten people. Right?
Like, have they gotten like great performance from modesty? Right? And it's
like, and I wonder, well, how much performance have we gotten from
rationality? Like, if I look at, say, imagine, like, I don't know, making bets
on whether the they're going to find the Higgs boson and like, making my own
meal supplements and like, it seems like concerning myself a lot with things
that are not that interesting or important, and wasting my time doing these
experiments, where like, I want to devote my life to like, the pursuits that
are of significance to me.
And I think, and I also, I already feel free to think about questions that
lots of other people have thought about, in fact that people have been
thinking about for thousands of years, right? And I have the– I feel like I
have some kind of hack to get around his problem, right? Where like, I think
it's OK for me to think about the question like, “What is being?” or
something. Where, like, so many people have been working on this question.
I mean, surely, if you can make progress, somebody already would have before
you and what kind of hubris, I guess I just have that hubris. So, I have some
like giant, you know, self-hack or something where I don't have to go through
all this. I can just ask the questions that interest me and pursue those
questions, because I think they're interesting and important. I don't need
like an excuse at this time to do it.
All right. So, at the end of the book, he talks about who the audience of his
book should be, and what could go wrong if it got into other hands.
So, in some sense, his target audience might be defined perhaps topologically
as the people who are too shy and relatively smart or confident. And if those
people who are relatively shy, but also confident if they were to try to think
for themselves more and try to do more things that would go well. And the
people who are not too shy, they don't benefit much from his advice to be less
shy. They might even push them too far in that direction. And the people who
are not smarter, confident enough, their trials won't go very well. And so, by
this description, well, if he selects his audience, right, he's guaranteed to
Right. There should be a toolstatic [0:45:51] [Phonetic] at the beginning of
the book. Right? And I would fail that like, on the first question and then I
would know, don't read this book, you're already going too far in the other
direction. So, there should also be a book for people like me, like, stop
thinking, you can answer all the big questions, like be a – be modest.
So, we started talking about this group, and then we went to talk about
But we come back to disagreement. So again, the key question is, when are you–
should you feel free to agree to disagree or foresee to disagree? And there is
this sort of modesty argument he outlines the very beginning of the book, and
then in more detail later on, which basically says, how could you know that
you are better than someone else at thinking about something? If they're
apriori equally qualified to you, and you come to different opinions, then
what justification do you have for picking yours over theirs?
And he is trying to resist that argument. And so, he offers a variety of
arguments like one that you mentioned that just merely abstractly trying to
disagree left is the wrong way to do it. You should do it organically and
authentically by thinking through all the details. But I think his strongest
arguments are the ones that just give particular cases and say, “Look, most
everybody disagrees with you on this, so you're really going to go with them?”
So, or like, for example, he says, “Look, a third of the time we're sleeping.
When we're sleeping, we don't usually know we're sleeping.”
You ask someone when they're sleeping, are you sleeping? They would say no.
And so, you at the moment can't be more than two-thirds confident that you're
not sleeping, because hey, you know, and of course, most of us are going to
say, “No, I'm a lot more than two-thirds confident that I'm not sleeping. I'm
pretty sure I'm awake.” And then he’ll say, “Well see that you're willing to
disagree.” Or he says, “You know…”
I mean that's how most people think. The way I think is, hey, Descartes gave a
really good argument that we should be pretty worried about that. And I'm
willing to engage that argument. And in fact, I hunt out people who are like,
you know, I want to keep looking and be skeptic or if they won’t, I’ll take
OK. But… But do you take that two-thirds position right now that you're
sleeping right now?
I think I'm awake right now.
OK. But at what confidence? Is it more than two-thirds?
So, like, this is where like, that's just that's not a natural way for me to
think about myself. Like, if you want to know, what would it take to push you
to assert that maybe you're sleeping right now? It wouldn't be that much.
Right? All you have to do is be like, I'm sure Descartes wrong. And I’ll be
So clearly, the sleeping example is not the one for you.
The next example, though, is God so, he says look, you know, clearly most
people in the world and certainly even more people in history that I believed
in God. So therefore, you…
OK. But I do believe in God. So that might not be the one for me either. So
maybe let's go to the next one.
Well, so, for those two are more persuasive examples for me. I mean, well, you
might even– so I might say, you know, the religion issue isn't is there some
abstract God out there somewhere doing something? I would think the religion
issue is more, do the people around us who claim to have some direct
connection with God, through prayer or sort of telling them things, is that a
real, actual communication phenomenon such that, that's an actual explanation
for a lot of things we see in the world is that people are talking to God, and
He's answering and doing things for them. So that's the position on which I
would be much more skeptical.
I'm happy to grant that in a vast universe, there could be vast powers out
there. And maybe they even created the universe, or even created our life on
earth or something. But the more skeptical claim, the one I have much more
trouble is the claim that all of this huge number of people, they claim that
their actual lives are being affected daily by direct communications and
Well, I mean, so those are two different claims, right? So like, I think my
life is affected by communication with God in that I tried to communicate with
God and if I would to stop doing that my life would be different. Right?
But you don’t get communications back that is you’re actually getting messages
Not that I know of. And I like, like God might well intervene in my life, but
I don't– I think that, like, it's very hard for me to understand God, or to
know what they do. So…
So, your position sounds much more reasonable to me. So, I'm not– but the
position that people think that they are seeing reliable evidence of
interventions. That's the position I would be skeptical about. And there are a
lot of people like that. I mean, even a majority of people will say…
Yeah, I mean, I guess I wouldn't dismiss the possibility of that, like, if I
met such a person. I've never had a conversation with such a person about that
very– no, I had one. Once, Liz Bruenig talked about this sort of like
religious experience that she had in a Night Owls that I did with her. I found
it pretty compelling, actually. But… so I guess like, I don't, I don't… but
it's not, it's not so much about intervention as a kind of like a kind of
awareness, right, like a kind of awareness of the presence of God.
Well, that would be caused in part by there being a, you know, making
themselves available to be aware of, right? I mean, they would participate in
that process somehow. Right? That would be an intervention.
But in any case, I guess I don't feel… to me, that's not like beyond the pale
But you still… I mean, I would still disagree in the sense of it to say…
You know, do most people have substantial evidence that they themselves are
seeing interventions of the sort? I would have to say, no. That sounds quite
Right. Or, like, yeah, I mean, I guess I feel less sure that that's so
unlikely. But I think let's take a thing where I just reflexively believe it
without thinking about. Those are always the best cases where you don't really
put much effort, like, I think that vaccines make you less likely to get
COVID, right? And I have put, like, no effort into thinking about that. I'm
just trusting, you know, people around me. But there's lots of people who
don't believe that.
So then does my belief go down, you know, my, like, slightly less credulous?
And I think the answer is no, not at all.
Or most people think their nation is substantially more moral than the rest of
the world. Or, that more justified in its actions on the international stage
in terms of morality or something. That would be another sort of claim that I
would find implausible that still most people do seem to believe it.
Right. I mean, but that's like a little bit like the claim that like most
people think like their family members are like better than other people,
which I'm very sympathetic to, because I think that too.
Right. But it's hard to believe it's true on average.
Right. Exactly. It's impossible that it’s true on average.
But I'm sympathetic to the claim. Like I get how that people think that.
People don't like, like, so I don't think people are crazy. I guess I just I
don't think people are crazy. I don't even think like, I guess I think it's
possible, like the vaccine case, I less know what to think about. The other
ones are more comfortable to me. Like, how can people…
But what we're looking for, for this rationality of disagreement issue is a
case where, in some sense, most people disagree with you, and you don't, you
can't really claim some authority relative to them, but still you're going to
disagree with them. And so, that would be the sleep or the evidence of God
example that they might give. And– but if you don't accept those that, of
course, you aren't finding counter examples to the modesty argument he calls
Right. But I look, I accept that with the vaccine case, I said, I think for
me, it's– the problem with this argument is that you slide back and forth
between disagreement and difference of opinion, in ways that, for me that is a
very, very significant difference. So, if I say most people have a difference
of opinion with me about something, like I'm virtually indifferent to that
then. But like, if somebody disagrees with me in the sense that I'm having a
conversation with them, and they think the opposite thing of what I think then
I have to take them seriously. And I have to argue with them, and I have to
convince them or be convinced or whatever, as long as the argument…
So, the theory that I worked on, is in terms of mutual awareness. So, it
doesn't require a conversation actually starts. But it requires that you each
be aware of the other person's opinion and mutually aware of these
awareness’s, etc. That's what common knowledge means. And so, the question
isn't, are you arguing at the moment with somebody who believes they have
evidence of God intervening in their lives? It’s, are you aware that they
think that or they aware that you think what you think? And is there a mutual
awareness such that if you started the conversation, you would both start off
with this mutual awareness? You wouldn't be surprised to find out what they
think and they wouldn't be surprised by the way you think, then you would, but
you would start discussing.
So let me just tell you something that’s not going to have a big impact on you
persuasively. But I read, years ago, I read some of this disagreement
literature, not the part that you contributed to, but… and my instinctual
response, and this is non-argument to my instinctual responses. All of these
people have framed the problem wrong. It's like people– it's like people going
out on a date but neither is willing to like, really give much of a signal
that they're interested in each other so they're just like doing these little
like signs. And then you're like, yeah, that's dating or something.
Like, there's such a thing as arguing. That's the real phenomenon, arguing,
where people are talking to each other. And the one guy says, “No, you're
wrong because of this.” That's the thing. But somehow people don't want to
study the actual thing. They want to study this weird, hypothetical case where
like, what if two people sort of knew of each other that they– but they didn't
actually have a conversation about it and how would they start out. But the
real phenomenon is arguing, and it's essentially stretched over time. So
people– it's important to the phenomenon that people change their positions
And they do, in fact, change like that is in every argument that I've ever
seen, they change in the sense that people say different things over time. So
their position is changing, its dynamics. It’s a dynamic process, right? And
so, you're talking about this static thing, disagree– and you're calling it
disagreement. There's this other thing, which is where our rationality really
sits, which is in this dynamic process of disagreement. And if you want to
understand it, that's the thing you're supposed to be talking about.
So, we’re running out of time for today, but we can at least set up the issue
here for our future conversation.
So, my understanding in talking to you previously is you have substantial
reservations about the usual decision theoretic framing. But the usual attempt
to understand argument, even in philosophy, as I understand it, is via
decision theory. That is, you imagine agents who have actions they need to
take. And they have beliefs and values. They combine the beliefs and values
into actions. Beliefs are updated on information. And beliefs can also be
updated on the information of talking to other people, arguing with them,
reading about arguments, and knowing what other people think, etc. And I have
contributed to that sort of literature using a sort of standard decision
theory framework to analyze how someone will update their beliefs upon
learning that someone else has a different belief.
So, you seem to be planting the flag of saying that whole framework is
misdirected, or mis– inappropriate for the analysis of disagreement.
Well, in that framework, you've talked about everything but conversation. It's
like, look, you could find out what other people think, not because you're
talking to them but somehow a little slip of paper falls back down that you
know… And oh, I learned what Robin thinks and now I could update that.
No, but the question is, when people talk, I mean, we do analyze conversation
in these theoretical frameworks. And conversation is analyzed as alternating
things people say, where each time they say something, they are picking some
symbols out of a space of possible symbols. And when you hear those particular
symbols you update on their mental state, having learned that they said the
symbol versus others.
So, it is the standard statistical inference framework in the context of
conversation. And in fact, there's a lot of analysis of conversation, exactly
that form like people have calculated how many bits per word there are in a
typical conversation that is, you know, how much can you anticipate what the
next word will be, out of this basic, all possible words, and, you know, that
framework is quite useful.
So, I mean, I think when I'm thinking about, when I'm speaking is like how to
persuade you, right? And what I can get you to think, but also what I can get
myself to think.
And, I have a kind of vague goal of being guided by the truth, while
maintaining the– as long as I can possibly maintain it, the stance of
disagreeing with you, as a tool to that end. So, I'm trying, as I said, that's
something I said in beginning I think it's when I'm trying to disagree with
you just is it rational for me to disagree with you, but the thing I'm doing
is disagreeing with you.
And so, it's not a question of like, I'm not trying to like, have, you know,
be figuring out my beliefs by bouncing them off of you. I'm like, and thus the
disagreement would be like an accidental upshot. I’m actively maintaining it.
I think we should stop soon. And there's something else I wanted to say. But
so, I'm going to let you respond and then I'm going to say everything I want
I want to acknowledge conversation is complicated, but we should just
distinguish sort of the beliefs you have in your head from the words you say
and the stances you take, and the conversational moves you make, and your
overall conversational strategies. Surely, conversational strategies consist
of anticipations of the moves you will make, the stances you will take, and
the responses other people will have to all that. But that's nonetheless
inconsistent with you're also having beliefs and that we could talk about
Yeah, I think the beliefs are [1:00:19] [Indiscernible] uninteresting. That is
people believe all sorts of random stuff that's like not very consistent. And
the more interesting and more stable thing is what they will say, or even
more, so what they will write.
So that's certainly in defiance of the usual decision theory framework that
says that when you go, when you leave the conversation to go make decisions,
it will be the beliefs you take with you. And that's what will influence who
you vote for, what job you take, and all those things. And so, when we're
trying to influence somebody's future actions, we're trying to influence the
beliefs they will have after the conversation. But anyway, your other point.
My other point is just I just wanted to note that you are, in this
conversation, this last conversation, like playing the role of systemic
policeman where you're like, “Look, we have a system, we have a usual framing
for dealing with this problem.” And you're trying to do the do-it-yourself
approach, so this recapitulates, the framework of the books.
I mean, I think that whenever you're defying the usual system, you should
notify people that facts, and I try to do that consistently. That is…
You often notify me of it.
Indeed. So, if you know that you are defying the usual wisdom on something, I
think you should still pursue your argument and try to persuade people, but
you should acknowledge that fact people, you should try to trick them into
thinking that what you say is the usual view if it's not. So yes, I want – and
that the fact that something is a contrary view, sort of sets up what kind of
conversational moves or arguments you might make regarding it. Now, you might
expect a criticism of the status quo, if you're going to be arguing for an
alternative, that'll be one of the things you expect.
So, there's a certain kind of, like, kind of like deferential act that you
think has to be done, where it's like, where you pay your respects to the
It's more of the fundamental argue– the fundamental argument rule, which is,
when you argue for something, you shall identify some of the best arguments
against it and tell people what those are too. That's the honest way to argue
for something is to both give the arguments for and against it. And so
certainly, if somebody is not aware that your view is the contrarian view,
then you should tell them that that's something that would inform them, they
might change their mind.
Not necessarily against it.
In most people's minds it is and therefore, again, you shouldn't necessarily
just give all the arguments you think are good arguments, you should also give
the arguments that other people might think are good that you might think
you're bad, and then tell them why they're bad,
Then you'd have to talk for a super long time before you let the other person
talk. So I think that's a bad plan. I think you should give your best argument
and then see what the other person says, and not think that you can anticipate
what would be the best arguments for them.
Well, it's about the timescale. Of course, you can give all arguments in a
short time, but in the course of a longer conversation you should at least
point to and give someone the chance to pursue it. So, I often like try to– at
the end of even my talks on prediction markets, I give a list of here's a
bunch of objections, you know, you can you can mention one of these if you
want, I don't go through them one by one, but I say, “Here are some common
objections, and you might pick one of these and ask about.”
Anyway, it's been nice talking.
Till we meet again.