Naive Rationality

Naive Rationality has classic failure modes.

Failing to take into account 2nd and 3rd effects
1. This often happens when those 2nd and 3rd effects are unknown to rationality but known to intuition
Long Term Thinking
1. Naive rationality leads to undervaluing poorly understood activity, (ex. Play / exploration) for maximizing expected value greedily in each moment
Lack of Complex Consequentialism
1. ‘Consistency’ demands that we choose either Deontology or Consequentialism, or some other framework for evaluating actions. But these systems all have value in particular contexts, and so should subsume one another. For example, effective rules often have to be clear (discrete) for psychological reasons, which leads to Deontology. The consequentialist should realize when that optimizes the outcome.
Destroying Trust Through Consequentialism
Disrespect for Intuition / Cultural Knowledge
1. What people mean when they say ‘wisdom’ is knowledge whose source is unknown, unknowable or forgotten. Passed from generations of Elders. Intuited through a distributed, parallel process in a mind that can’t be understood linearly.
Rational People Defect and Free Ride
1. There are many game-theoretic situations where cooperation is only enforced by social norms. In that context, a person relying on rationality instead of social imitation will defect (assuming that the long term ramifications for reputation, etc. are low)
Assuming Normal Distributions in Extremistan
What you see is all there is (relevant to Chesterton’s fence)
Believing that you’re rational leads to being overconfident in your understanding of the situation
Ignore the emotional side of thought / experience
Damning when emotions are the generators of action, rationalized
Appearances don’t matter, Environment doesn’t matter
This sense of invulnerability to emotional / cultural impact due to a belief in one’s own rationality
Chesterton’s Fence
The absence of understanding something is conflated with the misunderstood object being useless. An important instance of the overconfidence that comes out of naive rationality.
Two reformers find a fence in their way. The first doesn’t see the fence’s purpose, so he goes to knock it down. The second says that until the first reformer can see the purpose of the fence, he won’t allow him to knock it down.
Goodhart’s Law
Naive Rationality overfits to metrics that are available to a quantitative, empirical mind. But there’s much that science can’t touch and that is difficult to measure, and so when rationality is combined with something intuitive (say, ‘goodness’) it leads to EA / Singerism. In that case, lives saved is measurable.
"You know those trick questions which lure you into overly clean abstractions, and the wrong answer? All of rationalism is like that." - Sark

Chesterton’s Fence

Goodhart’s law (that which gets measured is no longer a meaningful measurement) combined with measuring something that used to be intuitive, leading to Singerism / EA

Naive rationality and not considering the long term

lack of complex consequentialism

Rational people defect and free ride

Assuming normal distributions in extremistan

What I see is all there is

By believing that you’re rational, you become overconfident in your understanding of the situation

Mean Reversion

Failing to consider Marginal Impact

Fixation

Naive rationality - a failure to take into account 2nd and 3rd order effects. Great example is SMBC’s satire on utilitarians and judging other people . Deontology often protects from 2nd and 3rd order effects. Wisdom looks like recognizing that these effects exist and behaving properly without necessarily being able to point to those effects.

Extremely naive rationality - appearances don’t matter, environment doesn’t matter

Ignores emotional side of thought / experience, destroying ability to be vulnerable (honest, open) when people have feelings

Source: Original Google Doc