Where "the Sequences" Are Wrong III

Reviewing the Sequence "The Craft And The Community"

[As in the other parts, I want to put a disclaimer here that I do not think the Sequences are less than about 70% correct. That number may change slightly, but I want to put this here in case someone were to see this and believe that I was trying to be more adversarial than I actually am. Things generally do not capture my attention well enough for me to study them in depth unless they are substantially useful and interesting material.]

I’m not going through the Sequences in order. For this part, I’m reviewing the final sequence, The Craft and the Community. This is where Eliezer¹ lays out his broad objectives for crafting a rationality community, and this project was laid down circa 2009.

Actually, one of the primary reasons for my starting this project (reviewing the Sequences) is my general observation that the rationality community today simultaneously has attractive and unattractive characteristics, which makes it look very good in theory but is not very fun in practice to be a member of. So I wanted to understand how much these things map back to Eliezer’s original blueprint.

Maybe I should give some attention in this piece to what I currently dislike about the community. I apologize that I am not able to be sufficiently non-adversarial here, about the community itself.

And that brings me to perhaps my least favorite thing about the rationality community, which is that it reacts to criticism very defensively. Which is odd, because it shouldn’t need to, if it is actually relatively rational compared to all other groups of people. I realize that the word “defensively” is kind of a euphemism, and taken literally, does not mean what I meant exactly. It’s quite okay to be literally defensive. What I dislike is when criticism is met with “they obviously hate us because they know we’re right and they’re wrong.”

This is why e/acc is an anti-tribe to rationality and EA, and not a sub-tribe—which I think would be a pareto-improvement to the current situation, if it was. And I am likely to consider this to remain a pareto-improvement well into the future, so I would also be likely to continue to push for that possibility to the extent it seems possible. I hope you can see that my adversarial stance is therefore not anti-rationality and not zero-sum—in favor of e/acc or critics of EA and rationality.

I want to emphasize that this degree of defensiveness actually does seem at odds with parts of Eliezer’s writing, such as a piece that I do like, Bayesians vs. Barbarians. I strongly suspect that it requires some degree of expecting “barbarians” to have an advantage in conflict for the tribe to have this much latent hostility to competing tribes. Granted, that piece—if emphasized to the degree I think it ought to be—is also kind of anti-Orthogonality Thesis to some extent, in my opinion.

It would seem to me that if you expect “barbarians” (defined as people who prefer conflict over cooperation, and are more incorrigible than corrigible) to have an advantage over their opposite types during conflict, then “rationality” would encourage being more barbarian-like.

My current position is that if the rationality community didn’t cling to the Orthogonality Thesis so strongly, it would be able to be less “barbarian” than it is now—unfortunately, I think that clinging to the basis for concern about AI risk does have a serious number of bullets to be bitten. One of these bullets is that “good guys with a gun do not necessarily beat bad guys with a gun, even in the long term, even in expectation over many instances, even if good guys tend to come from more advanced economies, etc., etc..”

Biting this bullet implies that you should be at least a tad more nervous when under attack by people who seem less rational and more evil than you are than you would otherwise be. But I personally find the values—rationality distinction a lot less clear and more fuzzy than most ordinary rationalists seem to.

Eliezer argues in the Sequences that rationality is instrumental for lots of different tasks. I think this is of course true, as rationality is defined to be that which is instrumentally useful for an arbitrary task. But he also, importantly, emphasizes this point to make it clear that any particular objective is not necessarily implied by one rational agent having it, to be important to another.

Therefore—I speculate—when the rationality community hears an argument from someone who claims that their set of objectives is rationally implied to be important for them to follow too, the community may react with hostility (moral disapproval) as opposed to (and instead of) rationalistic disagreement. The community, given that it believes that rationality does not necessarily specify which values one should have, may interpret such arguments as sneaky attempts to inject different values into the community, and naturally has somewhat of an allergic reaction to such a thing.

Moving on now to the subject of this review—if I were to give my shortest, gut-level take on the entire thing, it would be that Eliezer mainly talks about all of the different ways that humans are great at banding together for terrible purposes, and terrible at banding together for great purposes, because rational and smart people are way more inclined to criticize each other rather than work together. While he makes lots of good observations about how people work, what he gets the most wrong is the degree to which people are actually motivated better by false (religious) things than true things.

The Craft And The Community — (1630 / 2200 = 74.1%)

And now, for the quotes!

Raising The Sanity Waterline — (50% agreement)

But religion also serves the function of an asphyxiated canary in a coal mine—religion is a sign, a symptom, of larger problems that don't go away just because someone loses their religion.

Don't fight religion directly, instead, raise the general sanity waterline such that religion basically disappears as a symptom.

Certain "symptoms" we hate and look ugly to us, and yet, playing whack-a-mole with them doesn't seem to work.

Maybe if we teach people all these assorted rationality techniques, religion will disappear.

That's what the dead canary, religion, is telling us: that the general sanity waterline is currently really ridiculously low. Even in the highest halls of science.

If we throw out that dead and rotting canary, then our mine may stink a bit less, but the sanity waterline may not rise much higher.

I think Eliezer—throughout most of the Sequences, including here—underestimates the importance of certain aspects of religion while overestimating how perniciously bad those things are.

This may sound like I’m saying “religion is good.” Not really. Most religious people I know are not in favor of having a religion that updates to conceivably anything over time.

And most irreligious people I know probably don’t think of it quite like that—Science updates over time, religion does not, they say.

A Sense That More Is Possible — (100% agreement)

Why aren't "rationalists" surrounded by a visible aura of formidability? Why aren't they found at the top level of every elite selected on any basis that has anything to do with thought?

Why do most "rationalists" just seem like ordinary people, perhaps of moderately above-average intelligence, with one more hobbyhorse to ride?

Well, these days I am not sure if this is quite the case anymore.

No, but it's easier to verify when you have hit someone. That's part of it, a highly central part.

It is implied that "rationality" is more difficult to test. I am not sure if this is true. Rationality is a self-referential wish to become better at doing things better, which is to say, to become better at thinking. It is currently a low-hanging fruit because not everyone expresses this wish, even if it would be technically possible.

But first, because people lack the sense that rationality is something that should be systematized and trained and tested like a martial art, that should have as much knowledge behind it as nuclear engineering, whose superstars should practice as hard as chess grandmasters, whose successful practitioners should be surrounded by an evident aura of awesome.

This is sort of like a self-unfulfilling complaint.

But overall I still agree with the sentiment.

Epistemic Viciousness — (60% agreement)

One thing that I remembered being in this essay, but, on a second reading, wasn't actually there, was the degeneration of martial arts after the decline of real fights—by which I mean, fights where people were really trying to hurt each other and someone occasionally got killed.

Discussing the vibes that various "schools" (like martial arts dojos) give off. Namely, things like how special and authoritative this particular (niche) school is, loyalty to the instructor that involves strict adherence to the instructor's particular philosophy, deference to historical masters, and the difficulty of actually testing these techniques "in the wild."

This produces a dynamic where the actual objective function being optimized over is how impressed students are.

This is actually kind of an interesting and important one. I think the lack of full agreement comes from how important and non-dangerous I consider “attractive” vibes to be relative to what Eliezer seems to think of them. That said, he is talking about “mixed” vibes here, and schools which seem to use some amount of “dark arts” techniques.

Schools Proliferating Without Evidence — (60% agreement)

This is about, well, schools of thought (particularly in psychology) which proliferated for some time without any evidence.

I should really post more some other time on all the sad things this says about our world; about how the essence of medicine, as recognized by society and the courts, is not a repertoire of procedures with statistical evidence for their healing effectiveness; but, rather, the right air of authority.

But the subject today is the proliferation of traditions in psychotherapy. So far as I can discern, this was the way you picked up prestige in the field—not by discovering an amazing new technique whose effectiveness could be experimentally verified and adopted by all; but, rather, by splitting off your own "school", supported by your charisma as founder, and by the good stories you told about all the reasons your techniques should work.

This chapter doesn't appear to give any hypotheses for why this happens to the degree it seems to happen—but is implied to be due to some degree of “Goodharting”.

I think that the reasons given in the last sentence of the quote are implied by Yudkowsky to be orthogonal to true success, whereas I think they are probably more like "symptoms" of merely being a young and not-yet-developed field.

Three Levels of Rationality Verification — (100% agreement)

So even just the basic step of trying to ground reputations in some realistic trial other than charisma and good stories, has tremendous positive effects on a whole field of endeavor.

Stay grounded.

Why Our Kind Can't Cooperate — (80% agreement)

This section is interesting and important so I have many quotes here:

So far as any of those donors knew, they were alone. And when they tuned in the next day, they discovered not thanks, but arguments for why they shouldn't have donated.

The criticisms, the justifications for not donating—only those were displayed proudly in the open.

How do things work on the Dark Side?

The respected leader speaks, and there comes a chorus of pure agreement: if there are any who harbor inward doubts, they keep them to themselves.

So all the individual members of the audience see this atmosphere of pure agreement, and they feel more confident in the ideas presented—even if they, personally, harbored inward doubts, why, everyone else seems to agree with it.

If anyone is still unpersuaded after that, they leave the group (or in some places, are executed)—and the remainder are more in agreement, and reinforce each other with less interference.

Yes, a group which can't tolerate disagreement is not rational. But if you tolerate only disagreement—if you tolerate disagreement but not agreement—then you also are not rational.

You're only willing to hear some honest thoughts, but not others. You are a dangerous half-a-rationalist.

Our culture puts all the emphasis on heroic disagreement and heroic defiance, and none on heroic agreement or heroic group consensus.

We signal our superior intelligence and our membership in the nonconformist community by inventing clever objections to others' arguments.

These are very important observations. I have my own thoughts on what causes this dynamic too, which are not completely in disagreement with this.

It doesn’t really mention anything more in-depth or sophisticated than what “our culture” “does wrong.” The last sentence among these quotes is the only place where he supposes any kind of mechanism for what’s going on.

Tolerate Tolerance — (90% agreement)

Don't punish people for refusing to punish others.

Cooperation is unstable, in both game theory and evolutionary biology, without some kind of punishment for defection. So it's one thing to subtract points off someone's reputation for mistakes they make themselves, directly.

But if you also look askance at someone for refusing to castigate a person or idea, then that is punishment of non-punishers, a far more dangerous idiom that can lock an equilibrium in place even if it's harmful to everyone involved.

I agree with this, I just don’t think it goes far enough, given what was described in the previous section. Rationalists often reward each other for disagreement, including punishing others, and I think that punishing others for non-punishment is not the full story for what creates the environment of heavy disagreement.

Your Price for Joining — (100% agreement)

...It seems to me that people in the atheist/libertarian/technophile/sf-fan/etcetera cluster often set their joining prices way way way too high.

Like a 50-way split Ultimatum game, where every one of 50 players demands at least 20% of the money.

But usually... I observe that people underestimate the costs of what they ask for, or perhaps just act on instinct, and set their prices way way way too high.

If the issue isn't worth your personally fixing by however much effort it takes, and it doesn't arise from outright bad faith, it's not worth refusing to contribute your efforts to a cause you deem worthwhile.

I mean, I believe him—what he says about this particular cluster of people.

Can Humanism Match Religion's Output? — (50% agreement)

Really, I suspect that what's going on here has less to do with the motivating power of eternal damnation, and a lot more to do with the motivating power of physically meeting other people who share your cause.

The power, in other words, of being physically present at church and having religious neighbors.

This seems to be somewhat pre-EA. EA has motivated people to do a bunch of stuff in the name of non-religious utilitarian goodness. But I also think it’s not quite clear if that had much to do with physical presence.

Church vs. Task Force — (50% agreement)

So without copycatting religion—without assuming that we must gather every Sunday morning in a building with stained-glass windows while the children dress up in formal clothes and listen to someone sing—let's consider how to fill the emotional gap, after religion stops being an option.

Sometimes people say "Well what else would we do, if we didn't spend all this money on these beautiful cathedrals and churches? We would only have boring, ugly buildings and stuff."

And I am inclined to see a kernel of truth in that statement. Why is it that non-religion has to look so bland and boring? Why can’t we copycat at least some of the good aesthetic stuff? The vibes, man?

Rationality: Common Interest of Many Causes — (100% agreement)

If you and they don't agree on everything—especially priorities—you have to be willing to agree to shut up about the disagreement.

(Except possibly in specialized venues, out of the way of the mainstream discourse, where such disagreements are explicitly prosecuted.)

This is one of those insights where you blink incredulously and then grasp how much sense it makes. The human brain can't grasp large stakes and people are not anything remotely like expected utility maximizers, and we are generally altruistic akrasics. Saying, "This is the best thing" doesn't add much motivation beyond "This is a cool thing". It just establishes a much higher burden of proof. And invites invidious motivation-sapping comparison to all other good things you know (perhaps threatening to diminish moral satisfaction already purchased).

Perhaps we should fight less about which cause is the highest priority, and be willing to lend support to causes which are aided by rationality, since they increase the overall numbers of rationalists.

Helpless Individuals — (25% agreement)

Now I'm sure that if the general public were in the habit of funding particular science by individual donations, a whole lotta money would be wasted on e.g. quantum gibberish—assuming that the general public somehow acquired the habit of funding science without changing any other facts about the people or the society.

But it's still an interesting point that Science manages to survive not because it is in our collective individual interest to see Science get done, but rather, because Science has fastened itself as a parasite onto the few forms of large organization that can exist in our world. There are plenty of other projects that simply fail to exist in the first place.

Basically this is just his observations about how large-scale society seems to work. It's pretty negative and pessimistic, he basically laments what seems to be that individuals are not capable of investing in any important projects on their own; They are merely taxed, and then those funds go to a parasitic "Science" organization, where members fight among themselves for allocation of funds.

I am actually not entirely sure what the dynamic he is proposing is, if any, or he is merely noting problems. Has a score of 83 with 88 votes on LessWrong, and it is very rare for him to have a ratio of < 1.0.

Did he explore this idea more since then, though? Maybe in Inadequate Equilibria?

Money: The Unit of Caring — (100% agreement)

To the extent that individuals fail to grasp this principle on a gut level, they may think that the use of money is somehow optional in the pursuit of things that merely seem morally desirable—as opposed to tasks like feeding ourselves, whose desirability seems to be treated oddly differently. This factor may be sufficient by itself to prevent us from pursuing our collective common interest in groups larger than 40 people.

A dollar is basically one utilon. We should probably think in those terms instead of, say, hours spent volunteering at the soup kitchen.

Yes, frugality is a virtue. Yes, spending money hurts. But in the end, if you are never willing to spend any units of caring, it means you don't care.

Purchase Fuzzies and Utilons Separately — (0% agreement)

He argues that fuzzies and utilons are different. At best, fuzzies are willpower-restorers, but they are still selfish.

But if you find that you are like me in this aspect—that selfish good deeds still work—then I recommend that you purchase warm fuzzies and utilons separately. Not at the same time.

Trying to do both at the same time just means that neither ends up done well. If status matters to you, purchase status separately too!

But the main lesson is that all three of these things—warm fuzzies, status, and expected utilons—can be bought far more efficiently when you buy separately, optimizing for only one thing at a time.

I disagreed with this section perhaps more than any other part of this sequence. It just isn’t a convincing case that fuzzies and utilons really are non-overlapping, and I am more inclined to think that they are.

I found that besides that background assumption being inserted in there, the section also lacked much in the way of supporting arguments for the claim that “Trying to do both at the same time just means that neither ends up done well.”

Bystander Apathy — (90% agreement)

I've mused a bit on the evolutionary psychology of the bystander effect. Suppose that in the ancestral environment, most people in your band were likely to be at least a little related to you - enough to be worth saving, if you were the only one who could do it. But if there are two others present, then the first person to act incurs a cost, while the other two both reap the genetic benefit of a partial relative being saved. Could there have been an arms race for who waited the longest?

It’s kind of speculative but I see nothing wrong with that per se, as long as one doesn’t update too hard on it.

Collective Apathy And The Internet — (90% agreement)

But mostly I just hand you an open, unsolved problem: make it possible / easier for groups of strangers to coalesce into an effective task force over the Internet, in defiance of the usual failure modes and the default reasons why this is a non-ancestral problem.

I didn’t have many comments on this one. I guess saying “hey, it’s really hard to make a task force over the internet” over the internet kind of does work, though.

Incremental Progress And The Valley — (80% agreement)

"No," you say, "I'm talking about how startup founders strike it rich by believing in themselves and their ideas more strongly than any reasonable person would. I'm talking about how religious people are happier—"

Ah. Well, here's the the thing: An incremental step in the direction of rationality, if the result is still irrational in other ways, does not have to yield incrementally more winning.

And if, as I hope, someone goes on to develop the art of fighting akrasia or achieving mental well-being as thoroughly as I have developed the art of answering impossible questions, I do fully expect that those who wrap themselves in their illusions will not begin to compete. Meanwhile—others may do better than I, if happiness is their dearest desire, for I myself have invested little effort here.

This has key, meaty content worth thinking a lot about.

I wonder if instead of incrementally moving in the direction of more rationality, one should instead move incrementally in the direction of more winning. I predict that what may happen if one did this is that one might acquire some behaviors, habits, or characteristics that would emphatically not be non-rational, but which could contain some things that irreligious atheists might dislike or have been “allergic” to at some point.

Bayesians Vs. Barbarians — (100% agreement)

Would rationalists be worse at war than barbarians? He thinks no, in principle. At least he hopes so. Nothing, in principle, to disagree with here.

Consistently, I believe that rational agents are capable of coordinating on group projects whenever the (expected probabilistic) outcome is better than it would be without such coordination.

A society of agents that use my kind of decision theory, and have common knowledge of this fact, will end up at Pareto optima instead of Nash equilibria.

If all rational agents agree that they are better off fighting than surrendering, they will fight the Barbarians rather than surrender.

Beware of Other-Optimizing — (75% agreement)

It actually did take me a while to understand. One of the critical events was when someone on the Board of the Institute Which May Not Be Named, told me that I didn't need a salary increase to keep up with inflation—because I could be spending substantially less money on food if I used an online coupon service. And I believed this, because it was a friend I trusted, and it was delivered in a tone of such confidence. So my girlfriend started trying to use the service, and a couple of weeks later she gave up.

It's just that your earnest personal advice, that amazing thing you've found to actually work by golly, is no more and no less likely to work for me than a random personal improvement blog post written by an intelligent author is likely to work for you.

With the first quoted example, I wonder if he may have interpreted some disguised non-cooperation as cooperative. People do, in fact, behave non-cooperatively quite often, when the incentives appear to them to point in that direction, but also tend to conceal it when the incentives point toward doing that simultaneously.

At least from what he’s written here, he does seem to have taken that advice as totally sincere. But let’s not blame him much here, and certainly not more than the board-member, because I believe this particular fact is actually quite difficult to discover.

But from the description, a board member was said to advise frugality to who would probably be one of the highest-paid members of the organization. And although I don’t think the board member’s move was particularly wise, I also see that they had probable incentives to saying what they did.

I find that things I agree with in the 70% to 90% range seem to contain the largest amount of “Woah! This is really ripe for investigating more into these questions!”

Practical Advice Backed By Deep Theories — (80% agreement)

But practical advice really, really does become a lot more powerful when it's backed up by concrete experimental results, causal accounts that are actually true, and math validly interpreted.

Deep theories are quite nice to have, yes. I don’t know if we always have that luxury though, and I also don’t know if anyone is particularly against having deep theories on hand, though.

The Sin of Underconfidence — (100% agreement)

I would put it this way: A hypothesis affords testing! If you don't know whether you'll win on a hard problem—then challenge your rationality to discover your current level. I don't usually hold with congratulating yourself on having tried—it seems like a bad mental habit to me—but surely not trying is even worse.

I learned the first form of this rule at a very early age, when I was practicing for a certain math test, and found that my score was going down with each practice test I took, and noticed going over the answer sheet that I had been penciling in the correct answers and erasing them. So I said to myself, "All right, this time I'm going to use the Force and act on instinct", and my score shot up to above what it had been in the beginning, and on the real test it was higher still. So that was how I learned that doubting yourself does not always make you stronger—especially if it interferes with your ability to be moved by good information, such as your math intuitions. (But I did need the test to tell me this!)

This almost toots my own horn, so I have basically nothing to say about this one.

Go Forth And Create The Art! — (50% agreement)

I suspect—you could even call it a guess—that there is a barrier to getting started, in this matter of rationality. Where by default, in the beginning, you don't have enough to build on.

Indeed so little that you don't have a clue that more exists, that there is an Art to be found. And if you do begin to sense that more is possible—then you may just instantaneously go wrong.

As David Stove observes—I'm not going to link it, because it deserves its own post—most "great thinkers" in philosophy, e.g. Hegel, are properly objects of pity.

That's what happens by default to anyone who sets out to develop the art of thinking; they develop fake answers.

Unless I am reading this wrong, this seems to say, “You’ll probably fail and it will be considered your fault and people will laugh at you, but, good luck!”

I half-agree because we should do it, but we don’t have to expect quite that much failure.

Conclusion

The point of this exercise is to map my disagreements with this Sequence to a model, or at least a set of background assumptions, which I can compare to my own. I am pretty sure I know what my model is and how it feels different from Eliezer's, but it's a tad harder to show that the latter's model is what I think it is.

That being said, in the pieces that I found myself disagreeing with the most, a couple of patterns are fairly discernable:

1. "Religion" is seen as a thing that has both desirable and undesirable characteristics, but Eliezer most likely finds more surface-features than I do to be undesirable (or at least correlated with undesirable traits).

2. It seems hard to draw people together under the banner of rationality compared to religion or other "cool" causes. He doesn't offer much of a solution to this problem.

3. Fuzzies and utilons are different, and rationalists (often being utilitarian) want to optimize for the latter, and not the former (at least not at the same time). I think this is quite debatable and has serious costs.

4. The "dark side" looms as a serious issue, and is more serious the more you consider things like "fuzzies", "vibes", "coolness", or the surface-features of religion, to be unnecessary, undesirable, or a net-distraction from the rationalist cause or utilitarian causes. These things can be “Goodharted”, which, as a term you may be familiar with already from the AI risk literature, means it may come at great cost to things we consider “actually” important.

I believe the dark side becomes less threatening the more you relax and allow yourself to be persuaded by coolness and vibes, all while keeping in mind everything which grounds you to reality will successfully do so. You can only be distracted by coolness for a relatively short period of time. Eventually, you will want to see it really work. If you are so, so worried about being persuaded not to care about real things, this feeling will keep you caring about real things.

I find much of the practice of rationality to consist in noticing that you have already noticed things, and to allow yourself to update on evidence that already felt like it pushed you in some way.

1. I may have not mentioned it before, but referring to Eliezer Yudkowsky by his first name is a common practice by people who write a lot around or in the rationality community. Jesus H. Christ is also usually referred to by his first name, so it’s not that weird.