Where "the Sequences" Are Right I

(Where "the Sequences" Are Wrong V): Map and Territory (1/2) - Predictably Wrong & Fake Beliefs

I’m starting from the very beginning.

Previous post: Where “the Sequences” Are Wrong IV

This is the first chapter of “Where ‘the Sequences’ Are Right”, which is the fifth chapter of “Where ‘the Sequences’ Are Wrong.” The reason I’m doing it this way is not from planning ahead. My initial goal was only to find things wrong with the Sequences, but after doing so I realized I had gotten into the groove of just being able to review the entire thing. Besides, I realized I also didn’t want to be completely unfair.

You could presumably start reading here, or whichever post is most up to date, because I will be keeping a running summary of how the review as a whole is shaping up.

My general scoring rubric is shaping up to be something like the following:

• From each post, deduct points from 100.

• If it’s blindingly wrong¹ in its general thrust, -50 points (rare).

• If it’s subtly wrong in its general thrust, or seems to imply something that seems kind of important but is left unaddressed, I generally will write the most commentary for this situation, and it’s usually -10 to -25 points.

• If it’s neither subtly nor blindingly wrong in its general thrust, but has a few points I single out for consideration, -5 to -10 points each.

• If something in it made me laugh, +10 points up to 100.

• If something in it (like an analogy, metaphor, or illustration) just seems really cool, +10 points up to 100.

• I usually pick out at least one block quote from each post, and probably have at least one thing to say about it. I have non-uniform amounts of commentary.

The meta-goal, or rather the intellectual goal, is to work backwards from the few things I’ve already found “blindingly wrong” (which are mainly things I’ve written commentary about in the first four chapters), to see if those things were arrived at by any of the suspected elements I might consider “subtly wrong.” This is the goal I find to be the most interesting and fun, but we will likely do no better than conjecture even by the time this project is over.

Each chapter of my reviews, I’ll give a status update on where we are so far on the intellectual goal. Here’s where we are roughly today, at the end of the following review:

• I mostly consider the “blindingly wrong” elements (in the whole Sequences) thus far to be:

  • The value fragility hypothesis.

  • What I perceive as a kind of stubborn “GOFAI-ist” attitude, which actually has kind of profound implications for rationality, and so I don’t want to dismiss it lightly.

  • Uneven rigor. This is mostly in the “subtly wrong” category, but sometimes it is not.

• My working conjecture at this moment in time is something like the following:

  • The “blindingly wrong” elements do have something in common, most likely.

  • It seems to be a sort of vague preference for precision and formal proof that’s required before we get to believe that it is safe to take certain actions.

  • This preference dominates what I would also somewhat vaguely describe as the “how”: Are formal proofs for showing us how to do what we want, or are they more for showing us that we are permitted to do what we want?

• And I also have something of a “null hypothesis”:

  • The “blindingly wrong” elements are rare, and certainly are by number of sentences. This is because they represented embryonic thoughts on issues of high emotional relevance. This causes them both to have stuck out to me as well as for them to become important cruxes later on.

I consider that preference posited to exist in my working conjecture to be validly deemed pessimistic. The reason for that is: There is already a fuzzy distinction between being permitted to do something and knowing how. If I don’t know how to do something, then I am functionally not permitted to do it. If I do know how to do something, then “permitted to do it” is not precisely the right phrase. The grey area is for when it seems like we know how to do something, but without precise guarantees that it will do exactly what we want.

Pessimism is self-defeating when applied fairly and uniformly, because one notices that it is actually secretly wildly optimistic about other things (e.g., asymmetric difficulty in a game theoretic situation, or about how much utility can be gained by doing absolutely nothing). When you correct for this by applying a fairly pessimistic prior everywhere that you can find, the result you get is that you are just ever-so-slightly optimistic on the whole.

Even if you do not agree with that, or are at least not convinced, you may still be interested to see how my investigation pans out (which would lend confirmatory evidence in either direction).

So we begin.

Map and Territory (1/2) - Predictably Wrong & Fake Beliefs

Scope Insensitivity - (90/100)

The book starts out with Yudkowsky's case for why this project should exist. I agree that this project should exist, even in hindsight.

Once upon a time, three groups of subjects were asked how much they would pay to save 2,000 / 20,000 / 200,000 migrating birds from drowning in uncovered oil ponds. The groups respectively answered $80, $78, and $88.1 This is scope insensitivity or scope neglect: the number of birds saved—the scope of the altruistic action—had little effect on willingness to pay.

What's being described in this post is completely described in the first paragraph. It's a powerful image, but, as of yet, we are only aware that a problem exists, but we may not know how widespread it is or how much "damage" it actually causes.

I personally wonder about what would happen if I were asked - out of the blue - if I wanted to save 10 birds, and how much I would pay for that. Awaiting my answer, the questioner takes whatever number I give for that and asks if I want to now save 100 birds, if I would be willing to pay 10x my previous number. Maybe I am. Now they ask if I'd pay 100x that to save 1000, and so on.

But it seems like there's probably a fixed number of dollars that I'd pay to help save birds of some minimum number. The dollar amount can't really exceed what I'm willing to spend on all altruistic things together. If the number of birds each dollar saves is too small as well, then I might take some other opportunity. So there merely being an "opportunity" to save birds doesn't exactly mean that I would pay for each incremental opportunity by the same amount.

I would want to compare this question to something that requires demand on your part, like a specific product. E.g., a cheeseburger. How much are you willing to pay for one cheeseburger right now? 2? 10? 100?

You can see that it probably falls off quickly.

That being said, the numbers given by each of the groups in the example are even less coherent than that.

The Martial Art of Rationality - (85/100)

I often use the metaphor that rationality is the martial art of mind.

This really sells the program.

Although I've retitled these reviews, I'm secretly doing the same thing I was before, which also included noting correct things.

And where there are systematic errors human brains tend to make—like an insensitivity to scope—rationality is about fixing those mistakes, or finding work-arounds.

Here's where Yudkowsky opens with his grand strategy. I don't think that this is not what rationality should be all about, but throughout these reviews I have more-or-less worked backwards towards things like this to identify a kind of systematic tendency to look for flaws in thinking, rather than for improved techniques (which I don't believe are the same thing).

I don't know if you can always fix a mistake by saying "don't do that." E.g., "don't be scope insensitive." For me, it's not actually perfectly clear that each group - and I note that I’ve now shifted to talking about the previous post - should have given a number in perfect multiples of 10 from one another. What it should be is obviously something better than what it was, but with more thought applied. If it was that simple, I could simply see that, immediately point to a concept in my mind pattern-matching to scope insensitivity, and say "oh, obviously turn that off!"

IIRC, Yudkowsky at a later point says "reversed stupidity is not intelligence" which may in fact be the same thing I'm saying here. Is there an inconsistency? We have yet to determine that.

With their aid, we may be able to see more clearly the muscles of our brains, the fingers of thought as they move. We have a shared vocabulary in which to describe problems and solutions. Humanity may finally be ready to synthesize the martial art of mind: to refine, share, systematize, and pass on techniques of personal rationality.

I think that this overplays how many techniques will be valid for general reasoning and underplays how many techniques will be require domain knowledge (not that knowing what domain you're in wouldn't be considered general).

This was still before prediction markets were realized. As of today, we now have some functioning prediction markets that we didn't have before. How well do these help downstream tasks and projects? That still remains to be seen.

Availability - (85/100)

Burton et al. report that when dams and levees are built, they reduce the frequency of floods, and thus apparently create a false sense of security, leading to reduced precautions.2 While building dams decreases the frequency of floods, damage per flood is afterward so much greater that average yearly damage increases.

It would to check on how true / replicable this is. This seems like kind of an exaggerated or cherry-picked tale.

I don’t mind learning about a bias, though. It’s obviously not a bad idea to look out for them. It makes sense that this bias would exist, although I would think that this bias is one of the most amenable to learning from experience.

What's a Bias? - (90/100)

“Cognitive biases” are those obstacles to truth which are produced, not by the cost of information, nor by limited computing power, but by the shape of our own mental machinery.

It's interesting to say that biases are not caused by limited computing power or of limited information. In other words, he believes biases are errors in reasoning purely caused by the execution of poor mental maneuvers.

How I actually interpret this is that Yudkowsky is being ambitious with this sentence. He is being ambitious about his program and how much low-hanging fruit is there to be taken.

Burdensome Details - (85/100)

It might also have helped the forecasters to think about possible reasons why the US and Soviet Union would suspend diplomatic relations. The scenario is not “The US and Soviet Union suddenly suspend diplomatic relations for no reason,” but “The US and Soviet Union suspend diplomatic relations for any reason.”

Biases are generally ways that people reason about a problem incorrectly that are often correlated together. We don't all think about things wrongly in completely random ways. That being said, let's take a look at this one example, again.

I continue to think that the subtle thing nagging at me is really here. With the conjunction fallacy, people assign higher probability to "A & B" than to "A". But if I asked someone the general, abstract question about probability, "is p(A) higher or lower than p(A & B)?", my guess is that many people would either know this, or could reason about it by themselves more successfully than they would when given a word problem about geopolitical events.

My method at arriving at this hypothesis is by attempting to empathize with people making this particular mistake. I'm not guaranteed to get to the right conclusion about this either, but, when I try to simulate myself making this mistake (because real instances of myself actually making the conjunction fallacy proper don't come to mind easily), it is often for something close to the reason alluded to in the quote above.

That is, the reason that I’d assign A & B higher probability than A is that I hadn't mapped the sentences to an “A & B” and an “A” properly. Rather, it gets mapped to something like “B --> A” and “A.” That is, what is the probability of (B implies A) compared to just A?

Let's say A = "Reagan provides support for unwed mothers" and B = "Reagan cuts federal support for local governments." People may imagine politicians being forced to negotiate deals constantly in order to get anything through.

Therefore, they might think that B implies A.

I'm used to taking lots of tests, writing answers down on paper or filling in bubbles, and I'm also used to being presented with many tricky "gotcha" questions. Therefore, in such a context, I would be aware that my test-givers were looking for a specific answer that relied on being very careful with logic - in particular, making sure to interpret what was being said literally and mapping English sentences to a logical structure.

I could be empathizing way too much. People by-and-large really don't know how to do math, nor how to map things to logical structures on which math could be applied, to gain more accurate information about something which is actually relevant to people who don't know how to do math. So we could teach them this math, or preach to them about it, right?

What Do We Mean By "Rationality"? - (95/100)

I mean two things:

1. Epistemic rationality: systematically improving the accuracy of your beliefs.

2. Instrumental rationality: systematically achieving your values.

In a footnote, he says:

2 The idea that rationality is about strictly privileging verbal reasoning over feelings is a case in point. Bayesian rationality applies to urges, hunches, perceptions, and wordless intuitions, not just to assertions.

That's all well and good, but I think there is still a subtle System-2 privileging thing happening here that probably couldn't have been addressed without knowing the answers to trickier questions about intelligence.

Planning Fallacy - (90/100)

I actually like this post - even as it seems to deliver a blow against "optimism" - for actually providing a very large scale example of a planning failure. From this, we can now see how rationality would actually make significant differences on real-world applications.

(I don’t believe this post must be taken as a blow against “optimism” in the meta-optimism sense.)

A similar finding is that experienced outsiders, who know less of the details, but who have relevant memory to draw upon, are often much less optimistic and much more accurate than the actual planners and implementers.

This must be why management consulting is such a popular industry.

It does, however, leave me with a sense of, “well, that kind of sucks.” We can’t expect ourselves to do anything properly unless we get some outsiders to give their okay on it?

Why Truth? - (90/100)

I tend to be suspicious of morality as a motivation for rationality, not because I reject the moral ideal, but because it invites certain kinds of trouble. It is too easy to acquire, as learned moral duties, modes of thinking that are dreadful missteps in the dance.

There are probably specific moral injunctions he’s thinking about, which I can easily infer on my own, but at some point it still might be good to talk about which ones those are.

Curiosity, pragmatism, and quasi-moral injunctions are all key to the rationalist project. Yet if you were to ask me which of these is most foundational, I would say: “curiosity.” I have my principles, and I have my plans, which may well tell me to look behind the curtain. But then, I also just really want to know. What will I see? The world has handed me a puzzle, and a solution feels tantalizingly close.

What are the alternatives?

Feeling Rational - (80/100)

So is rationality orthogonal to feeling? No; our emotions arise from our models of reality. If I believe that my dead brother has been discovered alive, I will be happy; if I wake up and realize it was a dream, I will be sad. P. C. Hodgell said: “That which can be destroyed by the truth should be.” My dreaming self’s happiness was opposed by truth. My sadness on waking is rational; there is no truth which destroys it.

Emotions are okay to have. This doesn't conflict with anything I believe - it could only be, at worst, less detailed than things I currently believe.

This, does, however, still kind of subtly place emotions running in parallel with rationality, and portrays them ever-so-slightly as reactive to things in one’s environment, to be interpreted mainly as motivators for subsequent actions. It kind of subtly leaves it at that. This project is about rationality, and feelings - while not something to subdue - are out of the scope of the project.

The Lens That Sees Its Flaws - (85/100)

Here is the secret of deliberate rationality—this whole process is not magic, and you can understand it. You can understand how you see your shoelaces. You can think about which sort of thinking processes will create beliefs which mirror reality, and which thinking processes will not.

I think this is what "System 2" probably actually is. But that's almost besides the point; I think the "lens that sees its own flaws" as a metaphor basically implies that it can't see the places that lack flaws as those are either translucent or doing the looking.

Making Beliefs Pay Rent (In Anticipated Experiences) - (95/100)

Or suppose your English professor teaches you that the famous writer Wulky Wilkinsen is actually a “retropositional author,” which you can tell because his books exhibit “alienated resublimation.” And perhaps your professor knows all this because their professor told them; but all they're able to say about resublimation is that it's characteristic of retropositional thought, and of retropositionality that it's marked by alienated resublimation. What does this mean you should expect from Wulky Wilkinsen’s books?

Above all, don’t ask what to believe—ask what to anticipate. Every question of belief should flow from a question of anticipation, and that question of anticipation should be the center of the inquiry. Every guess of belief should begin by flowing to a specific guess of anticipation, and should continue to pay rent in future anticipations. If a belief turns deadbeat, evict it.

This is one of the most famous and oft-cited portions of the Sequences.

Where does the “-5” point deduction come from? This is actually a -15 point deduction followed by a +10.

The minus -15 comes from that I would advise instead of “if a belief turns deadbeat, evict it” that one should “observe it mindfully, looking for why it likes to live in your mind, and why it seems to linger there at all, if it’s not serving the interests of experiential prediction” as this will overturn more interesting information.²

The +10 comes from it having interesting illustrations and overall having a lot of weight to it.

A Fable of Science and Politics - (100/100)

Ferris gasped involuntarily, frozen by sheer wonder and delight. Ferris’s eyes darted hungrily about, fastening on each sight in turn before moving reluctantly to the next; the blue sky, the white clouds, the vast unknown outside, full of places and things (and people?) that no Undergrounder had ever seen. “Oh, so that’s what color it is,” Ferris said, and went exploring.

It’s rather humorous because this quote is clearly the only reaction that Yudkowsky thinks is morally acceptable to have, if one’s entire world were to suddenly come crashing down around them.

Because it’s a memorable and colorful story, I have added back whatever points I would have deducted from it.

Belief In Belief - (95/100)

While I disagree with Dennett on some details and complications, I still think that Dennett’s notion of belief in belief is the key insight necessary to understand the dragon-claimant. But we need a wider concept of belief, not limited to verbal sentences. “Belief” should include unspoken anticipation-controllers. “Belief in belief” should include unspoken cognitive-behavior-guiders. It is not psychologically realistic to say, “The dragon-claimant does not believe there is a dragon in their garage; they believe it is beneficial to believe there is a dragon in their garage.” But it is realistic to say the dragon-claimant anticipates as if there is no dragon in their garage, and makes excuses as if they believed in the belief.

Have I ever believed in belief? Sure, I think so. And I do mean the specific sub-type contextually defined here. It’s another thing that’s fairly easy to identify and realize, “yeah, you’re right, that didn’t make sense, and I didn’t really enjoy doing it that much in the first place.”

Religion's Claim To Be Non-Disprovable - (100/100)

The people of Israel are wavering between Jehovah and Baal, so Elijah announces that he will conduct an experiment to settle it—quite a novel concept in those days! The priests of Baal will place their bull on an altar, and Elijah will place Jehovah’s bull on an altar, but neither will be allowed to start the fire; whichever God is real will call down fire on His sacrifice. The priests of Baal serve as control group for Elijah—the same wooden fuel, the same bull, and the same priests making invocations, but to a false god. Then Elijah pours water on his altar—ruining the experimental symmetry, but this was back in the early days—to signify deliberate acceptance of the burden of proof, like needing a 0.05 significance level. The fire comes down on Elijah’s altar, which is the experimental observation. The watching people of Israel shout “The Lord is God!”—peer review.

And then the people haul the 450 priests of Baal down to the river Kishon and slit their throats. This is stern, but necessary. You must firmly discard the falsified hypothesis, and do so swiftly, before it can generate excuses to protect itself. If the priests of Baal are allowed to survive, they will start babbling about how religion is a separate magisterium which can be neither proven nor disproven.

Essentially illustrates that there was never a time at which people didn't believe their religion was literally experimentally provable. It is only after being disproven that people retreat to such a claim.

It was also funny, so I gave it back enough points that would have otherwise been deducted from it.

Professing and Cheering- (90/100)

I once attended a panel on the topic, “Are science and religion compatible?” One of the women on the panel, a pagan, held forth interminably upon how she believed that the Earth had been created when a giant primordial cow was born into the primordial abyss, who licked a primordial god into existence, whose descendants killed a primordial giant and used its corpse to create the Earth, etc. The tale was long, and detailed, and more absurd than the Earth being supported on the back of a giant turtle. And the speaker clearly knew enough science to know this.

What should we do about people who think it’s okay to knowingly profess that they believe something incompatible with science, but who say that because they are knowingly professing it, this makes it compatible with science?

Belief As Attire - (80/100)

In terms of humanly realistic psychology, the Muslims who flew planes into the World Trade Center undoubtedly saw themselves as heroes defending truth, justice, and the Islamic Way from hideous alien monsters a la the movie Independence Day. Only a very inexperienced nerd, the sort of nerd who has no idea how non-nerds see the world, would say this out loud in an Alabama bar. It is not an American thing to say. The American thing to say is that the terrorists “hate our freedom” and that flying a plane into a building is a “cowardly act.” You cannot say the phrases “heroic self-sacrifice” and “suicide bomber” in the same sentence, even for the sake of accurately describing how the Enemy sees the world. The very concept of the courage and altruism of a suicide bomber is Enemy attire—you can tell, because the Enemy talks about it. The cowardice and sociopathy of a suicide bomber is American attire. There are no quote marks you can use to talk about how the Enemy sees the world; it would be like dressing up as a Nazi for Halloween.

I think the reason I am subtracting points here is because it seems to argue for more symmetry between belief systems as they would be internally experienced by adherents than I think is justified (and we would certainly hope that there is as little symmetry as possible, so that this makes the fruits of rationality hang even lower than otherwise).

I think that the terrorists probably did hate our freedom, but I am not sure if they were cowardly. I think that they probably felt from the inside like they were eradicating vermin, and that this is an extremely noticeably asymmetric feeling from what is normally considered morally acceptable in our culture.

I think that this makes it more difficult to reduce the strangeness of other cultures and religious beliefs to mere attire. I think that people can have distinctively different beliefs and ways of thinking as experienced from the inside, these do produce different anticipations, and these anticipations may be dramatically proven wrong, later.

Pretending To Be Wise - (90/100)

This I call “pretending to be Wise.” Of course there are many ways to try and signal wisdom. But trying to signal wisdom by refusing to make guesses—refusing to sum up evidence—refusing to pass judgment—refusing to take sides—staying above the fray and looking down with a lofty and condescending gaze—which is to say, signaling wisdom by saying and doing nothing—well, that I find particularly pretentious.

I agree with the sentiment of this post, but I also don’t think it usually gets applied reliably. We might just not be able to easily apply a principle about this in either direction.

Applause Lights - (75/100)

I think it means that you have said the word “democracy,” so the audience is supposed to cheer. It’s not so much a propositional statement or belief, as the equivalent of the “Applause” light that tells a studio audience when to clap.

The exchange he describes started out (not his fault) as a tad hostile, and this meant that things such as “applause lights” became a strategy that a person might find viable, unfortunately.

By the way, this was what the speaker said that I registered as being slightly hostile³:

The first scenario sounds like an editorial in Reason magazine, and the second sounds like a Hollywood movie plot.

The problem we have which I haven’t seen much discussed throughout the Sequences is that as soon as people argue, epistemics are damaged - which is noted already - but not symmetrically. When someone is trying to defeat their opponent, this is slightly different than someone trying to win.

This post’s general thrust is that there’s something wrong with being both positive and vague at the same time. Being both positive and vague can be slightly positive, but not negative.

The negative thing was that the other speaker attempted to say, in a sense, “Not Yudkowsky’s plan, my vaguely positive plan!”

Final Score (1600 / 1800 = 88.9)

1. You may assume of course that it is “blindingly wrong” with respect to me. I do think that what I currently believe is probably true.

2. I’m leaving a note that this could turn out to be a more important point than I make it seem right here.

3. There are actually several accounts given by Yudkowsky in the Sequences where someone was being hostile to him in a way that was misidentified because he has been trained to be polite and charitable, like we all are.