Map and Territory: Predictably Wrong

This post is part of my notes on Eliezer Yudkowsky’s Rationality: A-Z, known as The Sequences.

Preface

• Reading the theory is one thing, but practising it is the key
• Rational action is similarly more important than rational belief

Biases

When your method of learning about the world is biased, learning more may not help. Acquiring data can even consistently worsen a biased prediction.

“The map is not the territory” - the claim is that our beliefs (the “map”) do not accurately represent the real world (the “territory”).

Scope Insensitivity

One hypothesis for why people are predictably insensitive to scope is that they “purchase moral satisfaction”, ie they spend enough money to feel good and altruistic (and then hit diminishing returns).

Weber’s Law states: “the minimum increase of stimulus which will produce a perceptible increase of sensation is proportional to the pre-existent stimulus”. Eg if you are very hungry and a eat a little, you will feel a big increase in satiation, whereas if you are already full and eat a little, you will feel very little increase in satiation.

The Martial Art of Rationality

Rationality is like training for martial arts - anyone who has a working hand can make a fist, anyone who has a working brain can become more rational. Some might do it slower or faster, but everyone is capable of it.

Availability

Availability bias is when your beliefs are distorted by secondhand data (and often, severalhand data).

Availability gives rise to a kind of absurdity bias; ie one where “events that have never happened are not recalled, and hence deemed to have probability zero.”

What’s a Bias?

“Cognitive biases” are those obstacles to truth which are produced… by the shape of our own mental machinery.

Burdensome Details

The conjunction fallacy is when humans assign a higher probability to the proposition of the form “A and B” than to one of the propositions “A” or “B” in isolation, even though it is a theorem that conjunctions are never likelier than their conjuncts.

This works because adding extra details can make a given scenario sound more plausible, even if it makes the events concerned less probable.

To avoid this - be wary of the word “and”. Be so wary that it causes you to recoil and reconsider.

This one isn’t super intuitive to me - so be especially watchful. Notice when things are supersets of the other things, and realise that it is impossible for a superset to have greater probability than its subset.

What Do We Mean By “Rationality”?

1. Epistemic rationality: systematically improving the accuracy of your beliefs.
2. Instrumental rationality: systematically achieving your values.

Probability theory is the set of laws underlying rational belief.

Decision theory is the set of laws underlying rational action.

Planning Fallacy

When a group of students were asked for 50/75/99 percent probability levels for a date by which their projects were completed, only 45% of students finished their projects by the time of their 99% probability level.

Human’s usually use the inside view for planning - what are the special, unique features of this task, and how long will those features take to implement. Instead consider using the outside view - broadly, how long to tasks of this nature tend to take.

Why Truth?

Some might consider truth-seeking to be morally important, but Eliezer thinks there are practical issues with that. Pursuing truth is worthwhile for instrumental reasons, but curiosity is key.

Feeling Rational

Rather than “emotion” and “rationality” as being opposed, instead consider there as System 1 and System 2.

Emotions can be rational - when it rests on true beliefs (eg there is a hungry lion charging toward me, so I feel fear). They can be irrational when resting on mistake beliefs.

There’s nothing wrong with feeling things strongly - even (or perhaps especially) if you’re a rationalist.

Despite all my philosophy I am still embarrassed to confess strong emotions, and you’re probably uncomfortable hearing them. But I know, now, that it is rational to feel.

The Lens That Sees Its Flaws

There is no such thing as magic. You can observe the world around you and understand it.

Crucially, you can also observe and understand yourself. In this way you can think about your own thinking, and thus alter it.

If you observe an optical illusion, there is a mismatch between the map and the territory. But if you are familiar with optical illusions and think to yourself “ah, this must be one of those”, then you can appropriately alter your map in order to more accurately reflect the territory.