Title: LessWrong Wiki
Tags: cloze titleFirst

Q: 2-place and 1-place words
A: A 2-place word acts as a function of 2 arguments ("2-place function").  For example:
Sexiness: Admirer, Entity -> [0, ∞)
A 1-place word acts as a function of 1 argument ("1-place function").  For example:
Sexiness: Entity -> [0, ∞)
Treating a 2-place word as a 1-place word can result in silly mistakes like assuming that a bug-eyed monster would naturally want to carry off a girl in a torn dress.  People think as if sexiness is an inherent property of a sexy entity, without dependence on the admirer.
"Of course the bug-eyed monster will prefer human females to its own kind," says the artist (who we'll call Fred); "it can see that human females have soft, pleasant skin instead of slimy scales.  It may be an alien, but it's not stupid - why are you expecting it to make such a basic mistake about sexiness?"

Q: A 2-place word acts as a function of 2 arguments ("2-place function").  For example:
Sexiness: Admirer, Entity -> [0, ∞)
A 1-place word acts as a function of 1 argument ("1-place function").  For example:
Sexiness: Entity -> [0, ∞)
Treating a 2-place word as a 1-place word can result in silly mistakes like assuming that a bug-eyed monster would naturally want to carry off a girl in a torn dress.  People think as if sexiness is an inherent property of a sexy entity, without dependence on the admirer.
"Of course the bug-eyed monster will prefer human females to its own kind," says the artist (who we'll call Fred); "it can see that human females have soft, pleasant skin instead of slimy scales.  It may be an alien, but it's not stupid - why are you expecting it to make such a basic mistake about sexiness?"
A: 2-place and 1-place words

Q: A Human\'s Guide to Words
A: A  series on the use and abuse of words; why you often can't define a word any way you like; how human brains seem to process definitions. First introduces the Mind projection fallacy and the concept of how an algorithm feels from inside, which makes it a basic intro to key elements of the LW zeitgeist.
A guide to this sequence is available at 37 Ways That Words Can Be Wrong.

Q: A  series on the use and abuse of words; why you often can't define a word any way you like; how human brains seem to process definitions. First introduces the Mind projection fallacy and the concept of how an algorithm feels from inside, which makes it a basic intro to key elements of the LW zeitgeist.
A guide to this sequence is available at 37 Ways That Words Can Be Wrong.
A: A Human\'s Guide to Words

Q: A sense that more is possible
A: The theory and practice of human rationality is currently at an embryonic stage. Religion runs rampant in the streets; it is not widely known that there is a systematic discipline of proper choice and belief. Some aspiring rationalists have  a sense that more is possible: a belief that a much more advanced art of rationality is possible to humans, that simply has not yet been developed. The Less Wrong project itself is part of this endeavor.

Q: The theory and practice of human rationality is currently at an embryonic stage. Religion runs rampant in the streets; it is not widely known that there is a systematic discipline of proper choice and belief. Some aspiring rationalists have  [...]: a belief that a much more advanced art of rationality is possible to humans, that simply has not yet been developed. The Less Wrong project itself is part of this endeavor.
A: A sense that more is possible

Q: Absolute certainty
A: Possessing absolute certainty in a fact, or Bayesian probability of 1, isn't a good idea. Losing an epistemic bet made with absolute certainty corresponds to receiving infinite negative payoff, according to the logarithmic proper scoring rule.
The same principle applies to mathematical truths. Not possessing absolute certainty in math doesn't make the math itself uncertain, the same way that an uncertain map doesn't cause the territory to blur out. The world, and the math, are precise, while knowledge about them is incomplete.
The impossibility of justified absolute certainty is sometimes used as a rationalization for the fallacy of gray.

Q: Possessing [...] in a fact, or Bayesian probability of 1, isn't a good idea. Losing an epistemic bet made with [...] corresponds to receiving infinite negative payoff, according to the logarithmic proper scoring rule.
The same principle applies to mathematical truths. Not possessing [...] in math doesn't make the math itself uncertain, the same way that an uncertain map doesn't cause the territory to blur out. The world, and the math, are precise, while knowledge about them is incomplete.
The impossibility of justified [...] is sometimes used as a rationalization for the fallacy of gray.
A: Absolute certainty

Q: Absurdity heuristic
A: The absurdity heuristic classifies highly untypical situations as "absurd", or impossible. While normally very useful as a form of epistemic hygiene, allowing to detect nonsense, it suffers from the same problems as representativeness heuristic.
There is a number of situations where absurdity heuristic is wrong. A deep theory has to override the intuitive expectation. Where you don't expect intuition to construct an adequate model of reality, classifying an idea as impossible may be overconfident. The future is usually "absurd", although sometimes it's possible to rigorously infer low bounds on capabilities of the future, proving possible what is intuitively absurd.

Q: The [...] classifies highly untypical situations as "absurd", or impossible. While normally very useful as a form of epistemic hygiene, allowing to detect nonsense, it suffers from the same problems as representativeness heuristic.
There is a number of situations where [...] is wrong. A deep theory has to override the intuitive expectation. Where you don't expect intuition to construct an adequate model of reality, classifying an idea as impossible may be overconfident. The future is usually "absurd", although sometimes it's possible to rigorously infer low bounds on capabilities of the future, proving possible what is intuitively absurd.
A: Absurdity heuristic

Q: Adaptation executors
A: Individual organisms are best thought of as adaptation-executers rather than as fitness-maximizers.
—John Tooby and Leda Cosmides, The Psychological Foundations of Culture
A central principle of evolutionary biology in general, and evolutionary psychology in particular.
If we regarded human taste buds as trying to maximize fitness, we might expect that, say, humans fed a diet too high in calories and too low in micronutrients, would begin to find lettuce delicious, and cheeseburgers distasteful.  But it is better to regard taste buds as an executing adaptation - they are adapted to an ancestral environment in which calories, not micronutrients, were the limiting factor.  And now they are simply executing that adaptation - evolution operates on too slow a timescale to re-adapt to such a recent condition.
Evolution is ultimately just a historical-statistical macrofact about which ancestors did in fact reproduce.  These genes then execute again, as they did previously.  And so the behavior of the organism is often better interpreted in terms of what worked in the past, rather than what should work in the future.  The organism's genes are, in fact, the causal result of what worked in the past, and certainly not a causal result of the future.

Q: Individual organisms are best thought of as adaptation-executers rather than as fitness-maximizers.
—John Tooby and Leda Cosmides, The Psychological Foundations of Culture
A central principle of evolutionary biology in general, and evolutionary psychology in particular.
If we regarded human taste buds as trying to maximize fitness, we might expect that, say, humans fed a diet too high in calories and too low in micronutrients, would begin to find lettuce delicious, and cheeseburgers distasteful.  But it is better to regard taste buds as an executing adaptation - they are adapted to an ancestral environment in which calories, not micronutrients, were the limiting factor.  And now they are simply executing that adaptation - evolution operates on too slow a timescale to re-adapt to such a recent condition.
Evolution is ultimately just a historical-statistical macrofact about which ancestors did in fact reproduce.  These genes then execute again, as they did previously.  And so the behavior of the organism is often better interpreted in terms of what worked in the past, rather than what should work in the future.  The organism's genes are, in fact, the causal result of what worked in the past, and certainly not a causal result of the future.
A: Adaptation executors

Q: Adversarial process
A: An adversarial process is a form of truth-seeking or conflict resolution in which identifiable factions hold one-sided positions.
Adversarial processes seem to be at the core of the Western idea of "fairness", which encourages their use in potentially violent contexts as a basic means of conflict reduction.  They are especially common in politics, where they're encouraged by the current use of democratic voting systems.  Legal courts often employ adversarial processes to evaluate a defendant's alleged wrong-doing and decide appropriate restitution.  Legal ethical codes and standards of evidence are needed in order to ensure that the overall process approximates truth-seeking.
The operation of a prediction market may also be seen as a kind of adversarial process, in which bid and ask prices are constantly compared.
Some view Wikipedia as an adversarial process where alternating edits to an article seek to balance what is seen by each author as an extreme point of view.  Wikipedia's policy of a "neutral point of view" implicitly assumes that such adversarial edits will eventually converge to an equilibrium.

Q: An [...] is a form of truth-seeking or conflict resolution in which identifiable factions hold one-sided positions.
[...]es seem to be at the core of the Western idea of "fairness", which encourages their use in potentially violent contexts as a basic means of conflict reduction.  They are especially common in politics, where they're encouraged by the current use of democratic voting systems.  Legal courts often employ [...]es to evaluate a defendant's alleged wrong-doing and decide appropriate restitution.  Legal ethical codes and standards of evidence are needed in order to ensure that the overall process approximates truth-seeking.
The operation of a prediction market may also be seen as a kind of [...], in which bid and ask prices are constantly compared.
Some view Wikipedia as an [...] where alternating edits to an article seek to balance what is seen by each author as an extreme point of view.  Wikipedia's policy of a "neutral point of view" implicitly assumes that such adversarial edits will eventually converge to an equilibrium.
A: Adversarial process

Q: Affect heuristic
A: The affect heuristic is a principle for making fast, perceptual judgments based on subjective impressions of goodness/badness. It can cause people to use one positive (or negative) fact about an object/action to judge the likely positivity or negativity of other facts about that object/action.

Q: The [...] is a principle for making fast, perceptual judgments based on subjective impressions of goodness/badness. It can cause people to use one positive (or negative) fact about an object/action to judge the likely positivity or negativity of other facts about that object/action.
A: Affect heuristic

Q: Affective death spiral
A: An affective death spiral (or happy death spiral) occurs when positive attributes of a theory, person, or organization combine with the Halo effect in a feedback loop, resulting in the subject of the affective death spiral being held in higher and higher regard. In effect, every positive thing said about the subject results in more than one additional nice thing to say about the subject on average. This cascades like a nuclear chain reaction. This process creates theories that are believed for their own sake and organizations that exist solely to perpetuate themselves, especially when combined with the social dynamics of groupthink. Affective death spirals are thus a primary cause of cultishness.
The same process can also occur with negative beliefs instead of positive, leading to a death spiral of hate.

Q: An [...] (or happy death spiral) occurs when positive attributes of a theory, person, or organization combine with the Halo effect in a feedback loop, resulting in the subject of the [...] being held in higher and higher regard. In effect, every positive thing said about the subject results in more than one additional nice thing to say about the subject on average. This cascades like a nuclear chain reaction. This process creates theories that are believed for their own sake and organizations that exist solely to perpetuate themselves, especially when combined with the social dynamics of groupthink. [...]s are thus a primary cause of cultishness.
The same process can also occur with negative beliefs instead of positive, leading to a death spiral of hate.
A: Affective death spiral

Q: Akrasia
A: Akrasia is the state of acting against one's better judgment. Examples of akrasia include procrastination and inability to form strong cooperating communities.
Note that, for example, if you are procrastinating because it's not in your best interest to complete the task you are delaying, it is not a case of akrasia.

Q: [...] is the state of acting against one's better judgment. Examples of [...] include procrastination and inability to form strong cooperating communities.
Note that, for example, if you are procrastinating because it's not in your best interest to complete the task you are delaying, it is not a case of [...].
A: Akrasia

Q: Alief
A: Alief is an independent source of emotional reaction which can coexist with a contradictory belief. For example, the fear felt when a monster jumps out of the darkness in a scary movie is based on the alief that the monster is about to attack you, even though you believe that it cannot.

Q: [...] is an independent source of emotional reaction which can coexist with a contradictory belief. For example, the fear felt when a monster jumps out of the darkness in a scary movie is based on the [...] that the monster is about to attack you, even though you believe that it cannot.
A: Alief

Q: Alien values
A: There are no rules requiring minds to value life, liberty or the pursuit of happiness. An alien will have, in all probability, alien values. If an "alien" isn't evolved, the range of possible values increases even more, allowing such absurdities as a Paperclip maximizer. Creatures with alien values might as well value only non-sentient life, or they might spend all their time building heaps of prime numbers of rocks.

Q: There are no rules requiring minds to value life, liberty or the pursuit of happiness. An alien will have, in all probability, [...]. If an "alien" isn't evolved, the range of possible values increases even more, allowing such absurdities as a Paperclip maximizer. Creatures with [...] might as well value only non-sentient life, or they might spend all their time building heaps of prime numbers of rocks.
A: Alien values

Q: Evolution as alien god
A: In the days before Darwin, people would look at the intricate complexity of life and blame it on a god - something with a mind.  Knowing the English sentence "evolution does not have a mind" doesn't automatically enable people to imagine a mindless, unintelligent designer.  People seem to visualize something more like a helpful spirit that works to improve species - whether they recite the phrase "evolution has no mind" or not.  They think that rabbits mate in order to "propagate their species", or wonder if evolution plans to carry on working through nonbiology...
People don't stop anthropomorphizing just because you tell them to.
To overcome the metaphor of the helpful spirit, one approach is to consider what kind of god evolution would be if it were a god - what sort of god takes a million years to invent a new complex design, what kind of god would put the retina in backwards, what kind of god would let elephants starve to death once they wore out their last set of teeth; and yet still managed to design a human brain, however poorly...
It would be a tremendously powerful, unbelievably stupid, ridiculously slow, and utterly uncaring god; a god monomaniacally focused on the relative fitness of genes within a species; a god whose attention was completely separated and working at cross-purposes in rabbits and wolves...
...in short, less like Jehovah, and more like H. P. Lovecraft's Azathoth, the blind idiot God burbling chaotically at the center of everything, surrounded by the thin monotonous piping of flutes.
And this is the process responsible for the creation of all biological life on Earth, and you in particular, so you should probably get to know it in some detail.

Q: In the days before Darwin, people would look at the intricate complexity of life and blame it on a god - something with a mind.  Knowing the English sentence "evolution does not have a mind" doesn't automatically enable people to imagine a mindless, unintelligent designer.  People seem to visualize something more like a helpful spirit that works to improve species - whether they recite the phrase "evolution has no mind" or not.  They think that rabbits mate in order to "propagate their species", or wonder if evolution plans to carry on working through nonbiology...
People don't stop anthropomorphizing just because you tell them to.
To overcome the metaphor of the helpful spirit, one approach is to consider what kind of god evolution would be if it were a god - what sort of god takes a million years to invent a new complex design, what kind of god would put the retina in backwards, what kind of god would let elephants starve to death once they wore out their last set of teeth; and yet still managed to design a human brain, however poorly...
It would be a tremendously powerful, unbelievably stupid, ridiculously slow, and utterly uncaring god; a god monomaniacally focused on the relative fitness of genes within a species; a god whose attention was completely separated and working at cross-purposes in rabbits and wolves...
...in short, less like Jehovah, and more like H. P. Lovecraft's Azathoth, the blind idiot God burbling chaotically at the center of everything, surrounded by the thin monotonous piping of flutes.
And this is the process responsible for the creation of all biological life on Earth, and you in particular, so you should probably get to know it in some detail.
A: Evolution as alien god

Q: Altruism
A: Actions undertaken for the benefit of other people. If you do something to feel good about helping people, or even to be a better person in some spiritual sense, it isn't truly altruism.

Q: Actions undertaken for the benefit of other people. If you do something to feel good about helping people, or even to be a better person in some spiritual sense, it isn't truly [...].
A: Altruism

Q: Ambient decision theory
A: A variant of updateless decision theory that uses first order logic instead of mathematical intuition module (MIM), emphasizing the way an agent can control which mathematical structure a fixed definition defines, an aspect of UDT separate from its own emphasis on not making the mistake of updating away things one can still acausally control.

Q: A variant of updateless decision theory that uses first order logic instead of mathematical intuition module (MIM), emphasizing the way an agent can control which mathematical structure a fixed definition defines, an aspect of UDT separate from its own emphasis on not making the mistake of updating away things one can still acausally control.
A: Ambient decision theory

Q: Amount of evidence
A: To a Bayesian, evidence is a quantitative concept. In principle, a hypothetical Bayesian superintelligence could say not only that the evidence supports a particular hypothesis, but by how much. (In practice the true math is usually intractable.) The more complicated or a priori improbable a hypothesis is, the more evidence you need just to justify it, or even just single it out of the amongst the mass of competing theories.
We often find it convenient to express the amount of evidence in terms of logarithms of odds, decibels if we use the base-10 logarithm, bits if we use the base-2.

Q: To a Bayesian, evidence is a quantitative concept. In principle, a hypothetical Bayesian superintelligence could say not only that the evidence supports a particular hypothesis, but by how much. (In practice the true math is usually intractable.) The more complicated or a priori improbable a hypothesis is, the more evidence you need just to justify it, or even just single it out of the amongst the mass of competing theories.
We often find it convenient to express the [...] in terms of logarithms of odds, decibels if we use the base-10 logarithm, bits if we use the base-2.
A: Amount of evidence

Q: Anthropomorphism
A: Anthropomorphism is the error of attributing distinctly human characteristics to nonhuman processes. As creatures who evolved in a social context, we all have adaptations for making predictions about other humans by empathic inference. When trying to understand the behavior of other humans, it oftentimes is a helpful (and bias-correcting) heuristic to ask, "Well, what would I do in such a situation?" and let that be your prediction. This mode of prediction simply won't do, however, for things (and in this wide universe there are many) that don't share the detailed structure bequeathed on the human brain by evolution, although it is oftentimes tempting.

Q: [...] is the error of attributing distinctly human characteristics to nonhuman processes. As creatures who evolved in a social context, we all have adaptations for making predictions about other humans by empathic inference. When trying to understand the behavior of other humans, it oftentimes is a helpful (and bias-correcting) heuristic to ask, "Well, what would I do in such a situation?" and let that be your prediction. This mode of prediction simply won't do, however, for things (and in this wide universe there are many) that don't share the detailed structure bequeathed on the human brain by evolution, although it is oftentimes tempting.
A: Anthropomorphism

Q: Anti-epistemology
A: Anti-epistemology refers to bad explicit beliefs about rules of reasoning, usually developed in the course of protecting an existing false belief - false beliefs are opposed not only by true beliefs (that must then be obscured in turn) but also by good rules of systematic reasoning (which must then be denied).  The explicit defense of fallacy as a general rule of reasoning is anti-epistemology.
Anti-epistemology was formerly known as Dark Side Epistemology but this seems to have been an overly poetic term that led to confusion with the Dark Arts - "anti-epistemology" refers to bad explicit rules about what is good to believe, not to invalid rhetorical techniques for persuasion.
An example of anti-epistemology run riot would be New Ageism:  The concept of "faith" was originally developed by scriptural religions, but with organized teachings that put boundaries around what you were allowed to have faith in.  New Ageism kept the notion that unjustified belief was a noble thing, but dropped the restrictive scriptures.  So now they have faith in unicorns, UFOs, homeopathy, pyramid power...

Q: [...] refers to bad explicit beliefs about rules of reasoning, usually developed in the course of protecting an existing false belief - false beliefs are opposed not only by true beliefs (that must then be obscured in turn) but also by good rules of systematic reasoning (which must then be denied).  The explicit defense of fallacy as a general rule of reasoning is [...].
[...] was formerly known as Dark Side Epistemology but this seems to have been an overly poetic term that led to confusion with the Dark Arts - "[...]" refers to bad explicit rules about what is good to believe, not to invalid rhetorical techniques for persuasion.
An example of [...] run riot would be New Ageism:  The concept of "faith" was originally developed by scriptural religions, but with organized teachings that put boundaries around what you were allowed to have faith in.  New Ageism kept the notion that unjustified belief was a noble thing, but dropped the restrictive scriptures.  So now they have faith in unicorns, UFOs, homeopathy, pyramid power...
A: Anti-epistemology

Q: Antiprediction
A: An antiprediction is a statement of confidence in an event that sounds startling, but actually isn't far from a maxentropy prior. Antiprediction fights the tendency of people to clutch to the last possibility, even where it's completely implausible that a certain event will happen, to a point where thinking about it becomes counterproductive or otherwise distorts the decision-making process. Just as prediction simplifies reasoning by allowing to assume that an event will occur and only planning for the case where it does occur, antiprediction simplifies the reasoning by allowing to completely disregard the case where the event occurs.
The reasons for keeping a hypothesis salient despite its unlikelihood are many; it may be emotional attachment, availability created either by experience or problem framing, or any other cognitive bias.
Often, it is sufficient to see the privileged possibility as but one among many equivalently probable events, and use maximum entropy principle to divide probability equally among them, leaving each very little. For example, each of the millions of possible lottery combinations are equally likely, and so any one person's ticket is very unlikely to come up: winning the lottery should be seen as one possibility among millions, not one of the two possibilities of "winning" and "not winning". Instrumentally, it's better to believe that winning is impossible, than that it's likely, if the actual probability is very low.

Q: An [...] is a statement of confidence in an event that sounds startling, but actually isn't far from a maxentropy prior. [...] fights the tendency of people to clutch to the last possibility, even where it's completely implausible that a certain event will happen, to a point where thinking about it becomes counterproductive or otherwise distorts the decision-making process. Just as prediction simplifies reasoning by allowing to assume that an event will occur and only planning for the case where it does occur, [...] simplifies the reasoning by allowing to completely disregard the case where the event occurs.
The reasons for keeping a hypothesis salient despite its unlikelihood are many; it may be emotional attachment, availability created either by experience or problem framing, or any other cognitive bias.
Often, it is sufficient to see the privileged possibility as but one among many equivalently probable events, and use maximum entropy principle to divide probability equally among them, leaving each very little. For example, each of the millions of possible lottery combinations are equally likely, and so any one person's ticket is very unlikely to come up: winning the lottery should be seen as one possibility among millions, not one of the two possibilities of "winning" and "not winning". Instrumentally, it's better to believe that winning is impossible, than that it's likely, if the actual probability is very low.
A: Antiprediction

Q: Applause light
A: We believe we are already within a democratic system.  Some factors are still missing, like the expression of the people's will. - Roberto Eduardo Viola
The substance of a democracy is the specific mechanism that resolves policy conflicts.  If all groups had the same preferred policies, there would be no need for democracy - we would automatically cooperate.  The resolution process can be a direct majority vote, or an elected legislature, or even a voter-sensitive behavior of an AI, but it has to be something.  What does it mean to call for a "democratic" solution if you don't have a conflict-resolution mechanism in mind?
I think it means that you have said the word "democracy", so the audience is supposed to cheer.  It's not so much a propositional statement, as the equivalent of the "Applause" light that tells a studio audience when to clap.

Q: We believe we are already within a democratic system.  Some factors are still missing, like the expression of the people's will. - Roberto Eduardo Viola
The substance of a democracy is the specific mechanism that resolves policy conflicts.  If all groups had the same preferred policies, there would be no need for democracy - we would automatically cooperate.  The resolution process can be a direct majority vote, or an elected legislature, or even a voter-sensitive behavior of an AI, but it has to be something.  What does it mean to call for a "democratic" solution if you don't have a conflict-resolution mechanism in mind?
I think it means that you have said the word "democracy", so the audience is supposed to cheer.  It's not so much a propositional statement, as the equivalent of the "Applause" light that tells a studio audience when to clap.
A: Applause light

Q: Arguing by analogy
A: Arguing by analogy is arguing that since things are alike in some ways, they will probably be alike in others. While careful application of argument by analogy can be a powerful tool, there are limits to the method after which it breaks down.

Q: [...] is arguing that since things are alike in some ways, they will probably be alike in others. While careful application of argument by analogy can be a powerful tool, there are limits to the method after which it breaks down.
A: Arguing by analogy

Q: Arguing by definition
A: Reality already is what it is; as aspiring rationalists, our task in thought and speech is to map it. You can't coerce reality by redefining a word. The fact that you've defined a word in a particular way can't teach you anything about the world, only observations can. Therefore arguing by definition is futile. Worse still, the dictionary definition of a word doesn't even begin to capture the rich cognitive content that actually constitutes its meaning; the dictionary, defining a word in terms of other words, can only serve as a helpful pointer to what most people mean when they use a word. Better to keep the focus on the facts of the matter and trying to understand what your interlocutor is trying to communicate, then to get lost in a pointless discussion of definitions, bearing nothing.

Q: Reality already is what it is; as aspiring rationalists, our task in thought and speech is to map it. You can't coerce reality by redefining a word. The fact that you've defined a word in a particular way can't teach you anything about the world, only observations can. Therefore [...] is futile. Worse still, the dictionary definition of a word doesn't even begin to capture the rich cognitive content that actually constitutes its meaning; the dictionary, defining a word in terms of other words, can only serve as a helpful pointer to what most people mean when they use a word. Better to keep the focus on the facts of the matter and trying to understand what your interlocutor is trying to communicate, then to get lost in a pointless discussion of definitions, bearing nothing.
A: Arguing by definition

Q: Arguments as soldiers
A: Arguments are soldiers.  Once you know which side you're on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it's like stabbing your soldiers in the back - providing aid and comfort to the enemy.  People who would be level-headed about evenhandedly weighing all sides of an issue in their professional life as scientists, can suddenly turn into slogan-chanting zombies when there's a Blue or Green position on an issue.
—Policy Debates Should Not Appear One-Sided
Arguments as soldiers is an alternate label for a conceptual metaphor which is more commonly stated as "Argument is War/Battle".  This metaphor is at the core of the adversarial system of debate which is widespread in politics, law and academia.  However, it is immature to hold on to this framework for private or deliberative decision making:
Identifying yourself with a side in a political debate may be detrimental to rational evaluation of arguments. Arguments get treated as soldiers, weapons to be used to defend your side of the debate, and to attack the other side. They are no longer instruments of the truth.

Q: Arguments are soldiers.  Once you know which side you're on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it's like stabbing your soldiers in the back - providing aid and comfort to the enemy.  People who would be level-headed about evenhandedly weighing all sides of an issue in their professional life as scientists, can suddenly turn into slogan-chanting zombies when there's a Blue or Green position on an issue.
—Policy Debates Should Not Appear One-Sided
[...] is an alternate label for a conceptual metaphor which is more commonly stated as "Argument is War/Battle".  This metaphor is at the core of the adversarial system of debate which is widespread in politics, law and academia.  However, it is immature to hold on to this framework for private or deliberative decision making:
Identifying yourself with a side in a political debate may be detrimental to rational evaluation of arguments. Arguments get treated as soldiers, weapons to be used to defend your side of the debate, and to attack the other side. They are no longer instruments of the truth.
A: Arguments as soldiers

Q: Artificial general intelligence
A: An Artificial general intelligence, or AGI, is a machine capable of behaving intelligently over many domains. The term can be taken as a contrast to narrow AI, systems that do things that would be considered intelligent if a human were doing them, but that lack the sort of general, flexible learning ability that would let them tackle entirely new domains. Though modern computers have drastically more ability to calculate than humans, this does not mean that they are generally intelligent, as they have little ability to invent new problem-solving techniques, and their abilities are targeted in narrow domains.
Contents
1 AGIs and Humans
2 "AGI" as a design paradigm
3 Expected dates for the creation of AGI
4 Blog posts
5 References
6 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
AGIs and Humans
Directly comparing the performance of AI to human performance is often an instance of anthropomorphism. The internal workings of an AI need not resemble those of a human; an AGI could have a radically different set of capabilities than those we are used to seeing in our fellow humans. A powerful AGI capable of operating across many domains could achieve competency in any domain that exceeds that of any human.
The values of an AGI could also be distinctly alien to those of humans, in which case it won't see many human activities as worthwhile and would have no intention of exceeding human performance (according to the human valuation of performance). Comparing an AGI's preferences to those of humans, AGI are classified as Friendly and Unfriendly. An Unfriendly AGI would pose a large existential risk.
"AGI" as a design paradigm
The term "Artificial General Intelligence," introduced by Shane Legg and Mark Gubrud, is often used to refer more specifically to a design paradigm which mixes modules of different types: "neat" and "scruffy", symbolic and subsymbolic. Ben Goertzel is the researcher most commonly associated with this approach, but others, including Peter Voss, are also pursuing it. This design paradigm, though eclectic in adopting various techniques,  stands in contrast to other approaches to creating new kinds of artificial general intelligence (in the broader sense), including brain emulation, artificial evolution, Global Brain, and pure "neat" or "scruffy" AI.
Expected dates for the creation of AGI
Reasons for expecting an AGI's creation in the near future include the continuation of Moore's law, larger datasets for machine learning, progress in the field of neuroscience, increasing population and collaborative tools, and the massive incentives for its creation. A survey of experts taken at a 2011 Future of Humanity Institute conference on machine intelligence found a 50% confidence median estimate of 2050 for the creation of an AGI, and 90% confidence in 2150. A significant minority of the AGI community views the prospects of an intelligence explosion or the loss of control over an AGI very skeptically however.

Q: An [...], or AGI, is a machine capable of behaving intelligently over many domains. The term can be taken as a contrast to narrow AI, systems that do things that would be considered intelligent if a human were doing them, but that lack the sort of general, flexible learning ability that would let them tackle entirely new domains. Though modern computers have drastically more ability to calculate than humans, this does not mean that they are generally intelligent, as they have little ability to invent new problem-solving techniques, and their abilities are targeted in narrow domains.
Contents
1 AGIs and Humans
2 "AGI" as a design paradigm
3 Expected dates for the creation of AGI
4 Blog posts
5 References
6 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
AGIs and Humans
Directly comparing the performance of AI to human performance is often an instance of anthropomorphism. The internal workings of an AI need not resemble those of a human; an AGI could have a radically different set of capabilities than those we are used to seeing in our fellow humans. A powerful AGI capable of operating across many domains could achieve competency in any domain that exceeds that of any human.
The values of an AGI could also be distinctly alien to those of humans, in which case it won't see many human activities as worthwhile and would have no intention of exceeding human performance (according to the human valuation of performance). Comparing an AGI's preferences to those of humans, AGI are classified as Friendly and Unfriendly. An Unfriendly AGI would pose a large existential risk.
"AGI" as a design paradigm
The term "Artificial General Intelligence," introduced by Shane Legg and Mark Gubrud, is often used to refer more specifically to a design paradigm which mixes modules of different types: "neat" and "scruffy", symbolic and subsymbolic. Ben Goertzel is the researcher most commonly associated with this approach, but others, including Peter Voss, are also pursuing it. This design paradigm, though eclectic in adopting various techniques,  stands in contrast to other approaches to creating new kinds of [...] (in the broader sense), including brain emulation, artificial evolution, Global Brain, and pure "neat" or "scruffy" AI.
Expected dates for the creation of AGI
Reasons for expecting an AGI's creation in the near future include the continuation of Moore's law, larger datasets for machine learning, progress in the field of neuroscience, increasing population and collaborative tools, and the massive incentives for its creation. A survey of experts taken at a 2011 Future of Humanity Institute conference on machine intelligence found a 50% confidence median estimate of 2050 for the creation of an AGI, and 90% confidence in 2150. A significant minority of the AGI community views the prospects of an intelligence explosion or the loss of control over an AGI very skeptically however.
A: Artificial general intelligence

Q: Aumann\'s agreement theorem
A: Aumann's agreement theorem, roughly speaking, says that two agents acting rationally (in a certain precise sense) and with common knowledge of each other's beliefs cannot agree to disagree. More specifically, if two people are genuine Bayesians, share common priors, and have common knowledge of each other's current probability assignments, then they must have equal probability assignments.

Q: [...], roughly speaking, says that two agents acting rationally (in a certain precise sense) and with common knowledge of each other's beliefs cannot agree to disagree. More specifically, if two people are genuine Bayesians, share common priors, and have common knowledge of each other's current probability assignments, then they must have equal probability assignments.
A: Aumann\'s agreement theorem

Q: Availability heuristic
A: The availability heuristic judges the probability of events by the ease with which examples come to mind. Sometimes this heuristic serves us well, but the map is not the territory; the frequency with which concepts occur in your thoughts need not reflect the frequency with which they occur in reality. Undue salience, selective reporting, even subtle features of how the human brain stores and recalls memories can distort our perceptions about the probability of events. Because it's easier to recall words by their first letter, people judge words that begin with the letter r to be more frequent than words with r as their third lettter, even though in fact, the latter is more frequent. Or selective reporting by the media of dramatic tragedies makes them seem more frequent than more threatening albeit mundane risks.

Q: The [...] judges the probability of events by the ease with which examples come to mind. Sometimes this heuristic serves us well, but the map is not the territory; the frequency with which concepts occur in your thoughts need not reflect the frequency with which they occur in reality. Undue salience, selective reporting, even subtle features of how the human brain stores and recalls memories can distort our perceptions about the probability of events. Because it's easier to recall words by their first letter, people judge words that begin with the letter r to be more frequent than words with r as their third lettter, even though in fact, the latter is more frequent. Or selective reporting by the media of dramatic tragedies makes them seem more frequent than more threatening albeit mundane risks.
A: Availability heuristic

Q: Bayesian
A: The secret technical codeword that cognitive scientists use to mean "rational".  Bayesian probability theory is the math of epistemic rationality, Bayesian decision theory is the math of instrumental rationality.  Right up there with cognitive bias as an absolutely fundamental concept on Less Wrong.
Contents
1 Philosophy
2 Math
3 Other usages
4 Blog posts
5 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Philosophy
Classical statistics is a bucket of assorted methods; different "methods" may give different answers for whether, e.g., an experimental result is "statistically significant".  In contrast, as the famous Bayesian E. T. Jaynes emphasized, probability theory is math and its results are theorems, every theorem consistent with every other theorem; you cannot get two different results by doing the derivation two different ways.
So is the project of rationality solved?  Indeed not.  First, probability theory and decision theory are often too computationally expensive to run in practice - it wouldn't take a galaxy-sized computer, so much as an unphysical computer (much larger than the known universe).  And second, it's not always clear how the math applies - even in theory, let alone the practice.
But we do know that violations of Bayesianism - even "unavoidable" violations due to lack of computing power - carry a price; a family of theorems demonstrates that anyone who does not choose according to consistent probabilities can be made to accept combinations of bets that are sure losses, or reject bets that are sure wins (the Dutch Book arguments); similarly, Cox's Theorem and its extensions show that anyone who obeys various "common-sensical" constraints on their betting probabilities must be representable in standard probability theory.
In other words, Bayesianism isn't just a good idea - it's the law, and if you violate it, you'll pay some kind of price.
When cognitive psychologists identify a cognitive bias, they know it's an error by comparison to the Bayesian gold standard.
Math
(Needs to be fleshed out.)  For introductions see probability theory, decision theory, and this introduction to Bayes' theorem.  A widely lauded technical book on this subject is E. T. Jaynes's "Probability Theory: The Logic of Science".
Other usages
"Bayesian" in philosophical usage often describes someone who adheres to the Bayesian interpretation of probability, viewing probability as a level of certainty in a potential outcome or idea. This is in contrast to a frequentist who views probability as a representation of how frequently a particular outcome will occur over any number of trials.
The term "Bayesian" may also refer to an ideal rational agent implementing precise, perfect Bayesian probability theory and decision theory (see, for example, Aumann's agreement theorem).

Q: The secret technical codeword that cognitive scientists use to mean "rational".  [...] probability theory is the math of epistemic rationality, [...] decision theory is the math of instrumental rationality.  Right up there with cognitive bias as an absolutely fundamental concept on Less Wrong.
Contents
1 Philosophy
2 Math
3 Other usages
4 Blog posts
5 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Philosophy
Classical statistics is a bucket of assorted methods; different "methods" may give different answers for whether, e.g., an experimental result is "statistically significant".  In contrast, as the famous [...] E. T. Jaynes emphasized, probability theory is math and its results are theorems, every theorem consistent with every other theorem; you cannot get two different results by doing the derivation two different ways.
So is the project of rationality solved?  Indeed not.  First, probability theory and decision theory are often too computationally expensive to run in practice - it wouldn't take a galaxy-sized computer, so much as an unphysical computer (much larger than the known universe).  And second, it's not always clear how the math applies - even in theory, let alone the practice.
But we do know that violations of [...]ism - even "unavoidable" violations due to lack of computing power - carry a price; a family of theorems demonstrates that anyone who does not choose according to consistent probabilities can be made to accept combinations of bets that are sure losses, or reject bets that are sure wins (the Dutch Book arguments); similarly, Cox's Theorem and its extensions show that anyone who obeys various "common-sensical" constraints on their betting probabilities must be representable in standard probability theory.
In other words, [...]ism isn't just a good idea - it's the law, and if you violate it, you'll pay some kind of price.
When cognitive psychologists identify a cognitive bias, they know it's an error by comparison to the [...] gold standard.
Math
(Needs to be fleshed out.)  For introductions see probability theory, decision theory, and this introduction to Bayes' theorem.  A widely lauded technical book on this subject is E. T. Jaynes's "Probability Theory: The Logic of Science".
Other usages
"[...]" in philosophical usage often describes someone who adheres to the [...] interpretation of probability, viewing probability as a level of certainty in a potential outcome or idea. This is in contrast to a frequentist who views probability as a representation of how frequently a particular outcome will occur over any number of trials.
The term "[...]" may also refer to an ideal rational agent implementing precise, perfect [...] probability theory and decision theory (see, for example, Aumann's agreement theorem).
A: Bayesian

Q: Bayesian decision theory
A: Bayesian decision theory refers to decision theory which is informed by Bayesian probability. An agent operating under such a decision theory uses the concepts of Bayesian statistics to estimate the expected value of its actions, and update its expectations based on new information.
Computer algorithms such as those studied in the subject of Machine learning can use Bayesian methods explicitly, but it also has been observed that naturally evolved [systems] mirror these probabilistic methods when they adapt to an uncertain environment. What Less Wrong refers to as Rationality is an effort to make conscious thoughts a better approximation of Bayesian decision theory, in order to better understand the world and achieve one's goals.

Q: [...] refers to decision theory which is informed by Bayesian probability. An agent operating under such a decision theory uses the concepts of Bayesian statistics to estimate the expected value of its actions, and update its expectations based on new information.
Computer algorithms such as those studied in the subject of Machine learning can use Bayesian methods explicitly, but it also has been observed that naturally evolved [systems] mirror these probabilistic methods when they adapt to an uncertain environment. What Less Wrong refers to as Rationality is an effort to make conscious thoughts a better approximation of [...], in order to better understand the world and achieve one's goals.
A: Bayesian decision theory

Q: Bayesian probability
A: Bayesian probability represents a level of certainty relating to a potential outcome or idea. This is in contrast to a frequentist probability that represents the frequency with which a particular outcome will occur over any number of trials.
An event with Bayesian probability of .6 (or 60%) should be interpreted as stating "With confidence 60%, this event contains the true outcome", whereas a frequentist interpretation would view it as stating "Over 100 trials, we should observe event X approximately 60 times."
The difference is more apparent when discussing ideas. A frequentist will not assign probability to an idea; either it is true or false and it cannot be true 6 times out of 10.

Q: [...] represents a level of certainty relating to a potential outcome or idea. This is in contrast to a frequentist probability that represents the frequency with which a particular outcome will occur over any number of trials.
An event with [...] of .6 (or 60%) should be interpreted as stating "With confidence 60%, this event contains the true outcome", whereas a frequentist interpretation would view it as stating "Over 100 trials, we should observe event X approximately 60 times."
The difference is more apparent when discussing ideas. A frequentist will not assign probability to an idea; either it is true or false and it cannot be true 6 times out of 10.
A: Bayesian probability

Q: Beisutsukai
A: The beisutsukai are members of a fictional society, also known as the Bayesian Conspiracy, depicted in a series of short stories by Eliezer Yudkowsky.  Characters in the story include Brennan (usually the viewpoint character) and his teacher Jeffreyssai.
Derived from "beisu", a Japanese transliteration of Bayes (though more accurate Japanese pronunciation would be bei-zu, not bei-su), plus "tsukai", Japanese "user", therefore "Bayes-user".  A casual, shortened pronunciation would probably come out "bayes-tskai".

Q: The [...] are members of a fictional society, also known as the Bayesian Conspiracy, depicted in a series of short stories by Eliezer Yudkowsky.  Characters in the story include Brennan (usually the viewpoint character) and his teacher Jeffreyssai.
Derived from "beisu", a Japanese transliteration of Bayes (though more accurate Japanese pronunciation would be bei-zu, not bei-su), plus "tsukai", Japanese "user", therefore "Bayes-user".  A casual, shortened pronunciation would probably come out "bayes-tskai".
A: Beisutsukai

Q: Belief
A: Belief is the mental state in which an individual holds a proposition to be true. Beliefs are often metaphorically referred to as maps, and are considered valid to the extent that they correctly correspond to the truth. A person's knowledge is a subset of their beliefs, namely the beliefs that are also true and justified. Beliefs can be second-order, concerning propositions about other beliefs.

Q: [...] is the mental state in which an individual holds a proposition to be true. [...]s are often metaphorically referred to as maps, and are considered valid to the extent that they correctly correspond to the truth. A person's knowledge is a subset of their [...]s, namely the [...]s that are also true and justified. [...]s can be second-order, concerning propositions about other [...]s.
A: Belief

Q: Belief as attire
A: A belief promoted by identification with a group or other signaling concerns, not by how well it reflects the territory. An improper belief.

Q: A belief promoted by identification with a group or other signaling concerns, not by how well it reflects the territory. An improper belief.
A: Belief as attire

Q: Belief in belief
A: As Daniel Dennett observes, where it is difficult to believe a thing, it is often much easier to believe that you ought to believe it. What does it mean to believe that the Ultimate Cosmic Sky is both perfectly blue and perfectly green? The statement is confusing; it's not even clear what it would mean to believe it - what exactly would be believed, if you believed. You can much more easily believe that it is proper, that it is good and virtuous and beneficial, to believe that the Ultimate Cosmic Sky is both perfectly blue and perfectly green.  Dennett calls this "belief in belief".
—Belief in Belief
Belief in belief is a situation where a model of the world you claim and believe to have is at odds with a model of the world that explains your actions and drives your anticipation of experience. You are always ready to provide an excuse for the next experiment not supporting your stated belief, because you know the outcome of the experiment in advance, but have to rationalize it in terms of the stated belief. Belief in belief can be seen as a form of compartmentalization, which protects your actual model of the world from insane memes that survive as stated beliefs. Were you to really believe and not just believe in belief, the consequences of error would be much more severe.

Q: As Daniel Dennett observes, where it is difficult to believe a thing, it is often much easier to believe that you ought to believe it. What does it mean to believe that the Ultimate Cosmic Sky is both perfectly blue and perfectly green? The statement is confusing; it's not even clear what it would mean to believe it - what exactly would be believed, if you believed. You can much more easily believe that it is proper, that it is good and virtuous and beneficial, to believe that the Ultimate Cosmic Sky is both perfectly blue and perfectly green.  Dennett calls this "[...]".
—Belief in Belief
[...] is a situation where a model of the world you claim and believe to have is at odds with a model of the world that explains your actions and drives your anticipation of experience. You are always ready to provide an excuse for the next experiment not supporting your stated belief, because you know the outcome of the experiment in advance, but have to rationalize it in terms of the stated belief. [...] can be seen as a form of compartmentalization, which protects your actual model of the world from insane memes that survive as stated beliefs. Were you to really believe and not just believe in belief, the consequences of error would be much more severe.
A: Belief in belief

Q: Belief update
A: What you do to your beliefs, opinions and cognitive structure when new evidence comes along.
Cox's theorem says, roughly, that if your beliefs at any given time take the form of an assignment of a numerical "plausibility score" to every proposition, and if they satisfy a few plausible axioms, then your plausibilities must effectively be probabilities obeying the usual laws of probability theory, and your updating procedure must be the one implied by Bayes' theorem.

Q: What you do to your beliefs, opinions and cognitive structure when new evidence comes along.
Cox's theorem says, roughly, that if your beliefs at any given time take the form of an assignment of a numerical "plausibility score" to every proposition, and if they satisfy a few plausible axioms, then your plausibilities must effectively be probabilities obeying the usual laws of probability theory, and your updating procedure must be the one implied by Bayes' theorem.
A: Belief update

Q: Beliefs require observations
A: To form accurate beliefs about something, you really do have to observe it.  This can be viewed as a special case of the second law of thermodynamics, in fact, since "knowledge" is correlation of belief with reality, which is mutual information, which is a form of negentropy.

Q: To form accurate beliefs about something, you really do have to observe it.  This can be viewed as a special case of the second law of thermodynamics, in fact, since "knowledge" is correlation of belief with reality, which is mutual information, which is a form of negentropy.
A: Beliefs require observations

Q: Bias
A: As used here, the term "bias" or "cognitive bias" refers to a specific, predictable error pattern in the human mind. The heuristics and biases program in cognitive psychology has documented hundreds of reproducible errors - often huge errors. This continues to be a highly active area of investigation in cognitive psychology.
Understanding cognitive biases and trying to defend against their effects has been a basic theme of Less Wrong since the days it was part of Overcoming Bias.

Q: As used here, the term "[...]" or "cognitive [...]" refers to a specific, predictable error pattern in the human mind. The heuristics and [...]es program in cognitive psychology has documented hundreds of reproducible errors - often huge errors. This continues to be a highly active area of investigation in cognitive psychology.
Understanding cognitive [...]es and trying to defend against their effects has been a basic theme of Less Wrong since the days it was part of Overcoming [...].
A: Bias

Q: Bite the bullet
A: To bite the bullet is to accept the consequences of a hard choice, or unintuitive conclusions of a formal reasoning procedure.

Q: To [...] is to accept the consequences of a hard choice, or unintuitive conclusions of a formal reasoning procedure.
A: Bite the bullet

Q: Black swan
A: It is the most common way of trying to cope with novelty: by means of metaphors and analogies we try to link the new to the old, the novel to the familiar. Under sufficiently slow and gradual change, it works reasonably well; in the case of a sharp discontinuity, however, the method breaks down: though we may glorify it with the name "common sense", our past experience is no longer relevant, the analogies become too shallow, and the metaphors become more misleading than illuminating. This is the situation that is characteristic for the "radical" novelty.
—E.W. Dijkstra, On the cruelty of really teaching computer science
A black swan is a high-impact event that is hard to predict (but not necessarily of low probability). Also, an event that is not accounted for in a model, and therefore causes the model to break down when it occurs.
Considering some event a black swan doesn't give a leave to not assign any probabilities, since making decisions depending on the plausibility of such event is still equivalent to assigning probabilities that make the expected utility calculation give those decisions.

Q: It is the most common way of trying to cope with novelty: by means of metaphors and analogies we try to link the new to the old, the novel to the familiar. Under sufficiently slow and gradual change, it works reasonably well; in the case of a sharp discontinuity, however, the method breaks down: though we may glorify it with the name "common sense", our past experience is no longer relevant, the analogies become too shallow, and the metaphors become more misleading than illuminating. This is the situation that is characteristic for the "radical" novelty.
—E.W. Dijkstra, On the cruelty of really teaching computer science
A [...] is a high-impact event that is hard to predict (but not necessarily of low probability). Also, an event that is not accounted for in a model, and therefore causes the model to break down when it occurs.
Considering some event a [...] doesn't give a leave to not assign any probabilities, since making decisions depending on the plausibility of such event is still equivalent to assigning probabilities that make the expected utility calculation give those decisions.
A: Black swan

Q: Color politics
A: In articles at Overcoming Bias and Less Wrong, the words "Blues" and "Greens" are often used to refer to two opposing political factions.
The terms come from the names of chariot racing teams, that differed in nothing but the team colors, but rivalry of whose fans sometimes reached the level of gang wars.[1]  By definition, politics also deals with matters that people physically fight over in the real world -- or at least, matters that are to be enforced by the government's monopoly on violence.
Politics commonly involves an adversarial process, where factions usually identify with political positions, and use arguments as soldiers to defend their side.  When tempered by appropriate standards of evidence, rules of order and other safeguards, such a process may be the only way of introducing a modicum of deliberative truth seeking and other virtues of rationality into an inherently violent domain.  However, the dichotomies presented by the opposing sides are often false dilemmas, which can be shown by presenting third options.
For a variery of reasons, Less Wrong tries to avoid political disputes: see Mind-killer.

Q: In articles at Overcoming Bias and Less Wrong, the words "Blues" and "Greens" are often used to refer to two opposing political factions.
The terms come from the names of chariot racing teams, that differed in nothing but the team colors, but rivalry of whose fans sometimes reached the level of gang wars.[1]  By definition, politics also deals with matters that people physically fight over in the real world -- or at least, matters that are to be enforced by the government's monopoly on violence.
Politics commonly involves an adversarial process, where factions usually identify with political positions, and use arguments as soldiers to defend their side.  When tempered by appropriate standards of evidence, rules of order and other safeguards, such a process may be the only way of introducing a modicum of deliberative truth seeking and other virtues of rationality into an inherently violent domain.  However, the dichotomies presented by the opposing sides are often false dilemmas, which can be shown by presenting third options.
For a variery of reasons, Less Wrong tries to avoid political disputes: see Mind-killer.
A: Color politics

Q: Burch\'s law
A: I think people should have a right to be stupid and, if they have that right, the market's going to respond by supplying as much stupidity as can be sold.
—Greg Burch
While Burch was referring specifically to SUV's (which he believed to be badly designed) his point carries far more generally.  Examples include:
Lotteries, which take advantage of people's inability to understand small probabilities.
Cigarettes, which take advantage of the tendency to think about the possibility of lung cancer in far mode while thinking about potential social benefits of smoking in near mode.
A corollary of Burch's Law is that any bias should be regarded as a potential vulnerability whereby the market can trick one into buying something one doesn't really want.

Q: I think people should have a right to be stupid and, if they have that right, the market's going to respond by supplying as much stupidity as can be sold.
—Greg Burch
While Burch was referring specifically to SUV's (which he believed to be badly designed) his point carries far more generally.  Examples include:
Lotteries, which take advantage of people's inability to understand small probabilities.
Cigarettes, which take advantage of the tendency to think about the possibility of lung cancer in far mode while thinking about potential social benefits of smoking in near mode.
A corollary of Burch's Law is that any bias should be regarded as a potential vulnerability whereby the market can trick one into buying something one doesn't really want.
A: Burch\'s law

Q: Burdensome details
A: The conjunction rule of probability theory states that a conjunction (A and B) is necessarily less probable than one of the conjuncts alone (A). Adding more details to a theory may make it sound more plausible to human ears because of the representativeness heuristic, even as the story becomes normatively less probable, as burdensome details drive the probability of the conjunction down (this is known as conjunction fallacy). Any detail you add has to be pinned down by a sufficient amount of evidence; all the details you make no claim about can be summed over.

Q: The conjunction rule of probability theory states that a conjunction (A and B) is necessarily less probable than one of the conjuncts alone (A). Adding more details to a theory may make it sound more plausible to human ears because of the representativeness heuristic, even as the story becomes normatively less probable, as [...] drive the probability of the conjunction down (this is known as conjunction fallacy). Any detail you add has to be pinned down by a sufficient amount of evidence; all the details you make no claim about can be summed over.
A: Burdensome details

Q: Bystander effect
A: The bystander effect is a social psychological phenomenon in which individuals are less likely to offer help in an emergency situation when other people are present. The probability of help is inversely proportional to the number of bystanders. In other words, the greater the number of bystanders, the less likely it is that any one of them will help.

Q: The [...] is a social psychological phenomenon in which individuals are less likely to offer help in an emergency situation when other people are present. The probability of help is inversely proportional to the number of bystanders. In other words, the greater the number of bystanders, the less likely it is that any one of them will help.
A: Bystander effect

Q: Cached thought
A: A cached thought is an answer that was arrived at by recalling the old conclusion, rather than performing the reasoning from scratch. Cached thoughts can result in the maintenance of a position when evidence should force an update. Cached thoughts can also result in a lack of creative approaches to problem-solving if one repeats the same cached thoughts rather than constructing a new approach.
What is generally called common sense is more or less a collection of cached thoughts.

Q: A [...] is an answer that was arrived at by recalling the old conclusion, rather than performing the reasoning from scratch. [...]s can result in the maintenance of a position when evidence should force an update. [...]s can also result in a lack of creative approaches to problem-solving if one repeats the same [...]s rather than constructing a new approach.
What is generally called common sense is more or less a collection of [...]s.
A: Cached thought

Q: Causal Decision Theory
A: Causal Decision Theory is a branch of decision theory which advises an agent to take actions causally related to a desired outcome. It maximizes P(A>O), the utility of outcome O in the counterfactual in which action A is performed, which contrasts against Evidential Decision Theory.
The difference between Causal Decision Theory and Evidential Decision Theory is highlighted by CDT's correct answer to the Smoking lesion problem. However, problems with CDT are demonstrated by other scenarios, such as Newcomb's problem.
See also
Decision theory
Evidential Decision Theory

Q: [...] is a branch of decision theory which advises an agent to take actions causally related to a desired outcome. It maximizes P(A>O), the utility of outcome O in the counterfactual in which action A is performed, which contrasts against Evidential Decision Theory.
The difference between [...] and Evidential Decision Theory is highlighted by CDT's correct answer to the Smoking lesion problem. However, problems with CDT are demonstrated by other scenarios, such as Newcomb's problem.
See also
Decision theory
Evidential Decision Theory
A: Causal Decision Theory

Q: Causality
A: Causality refers to the relationship between an event (the cause) and a second event (the effect), where the second event is a direct consequence of the first.

Q: [...] refers to the relationship between an event (the cause) and a second event (the effect), where the second event is a direct consequence of the first.
A: Causality

Q: Challenging the Difficult
A: A sequence on how to do things that are difficult or "impossible".

Q: A sequence on how to do things that are difficult or "impossible".
A: Challenging the Difficult

Q: Chronophone
A: Suppose you could send messages back in time to Archimedes of Syracuse, using a chronophone which - to avoid transmitting anachronistic information - transmits the results of executing cognitive strategies, rather than words.  If you say "Women should have the vote", it comes out as "Install a tyrant of great personal virtue", because you repeated what your culture considers a wise form of political arrangement, and what comes out of the chronophone is the result of executing the same cognitive policy in Archimedes's era.
The chronophone won't transmit arguments you rationalize using your home culture's foreknowledge of the desired conclusion - it will substitute the result of executing that cognitive policy using Archimedes's culture's belief as the intended conclusion.  A basic principle of the chronophone is that if you say something considered obvious in your home culture, it comes out as something considered obvious in Archimedes's culture.
The challenge was to say something useful under this restriction.  This challenge is supposed to be difficult.  It's really hard to get somewhere when you don't already know your destination.  If there were some simple cognitive policy you could follow to spark moral and technological revolutions, without your home culture having advance knowledge of the destination, you could execute that cognitive policy today - which is what the whole parable is about!

Q: Suppose you could send messages back in time to Archimedes of Syracuse, using a [...] which - to avoid transmitting anachronistic information - transmits the results of executing cognitive strategies, rather than words.  If you say "Women should have the vote", it comes out as "Install a tyrant of great personal virtue", because you repeated what your culture considers a wise form of political arrangement, and what comes out of the [...] is the result of executing the same cognitive policy in Archimedes's era.
The [...] won't transmit arguments you rationalize using your home culture's foreknowledge of the desired conclusion - it will substitute the result of executing that cognitive policy using Archimedes's culture's belief as the intended conclusion.  A basic principle of the [...] is that if you say something considered obvious in your home culture, it comes out as something considered obvious in Archimedes's culture.
The challenge was to say something useful under this restriction.  This challenge is supposed to be difficult.  It's really hard to get somewhere when you don't already know your destination.  If there were some simple cognitive policy you could follow to spark moral and technological revolutions, without your home culture having advance knowledge of the destination, you could execute that cognitive policy today - which is what the whole parable is about!
A: Chronophone

Q: Cognitive style
A: Certain cognitive styles might tend to produce more accurate results. A common distinction between cognitive styles is that of foxes vs. hedgehogs, based on Isiah Berlin's famous essay.
Contents
1 Foxes and Hedgehogs
2 Blog posts
3 See also
4 Other resources
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Foxes and Hedgehogs
This common distinction comes primarily from the essay "The Hedgehog and the Fox" by Isiah Berlin, regarding the Russian author Leo Tolstoy's theory of history.
The title is a reference to a fragment attributed to the ancient Greek poet Archilochus: πόλλ' οἶδ' ἀλώπηξ, ἐχῖνος δ'ἓν μέγα ("The fox knows many things, but the hedgehog knows one big thing").
Berlin expands upon this idea to divide writers and thinkers into two categories: hedgehogs, who view the world through the lens of a single defining idea and foxes who draw on a wide variety of experiences and for whom the world cannot be boiled down to a single idea.
This distinction is used by Philip Tetlock in the book Expert Political Judgement, which concludes that foxes tend to be better calibrated and more accurate[1].

Q: Certain [...]s might tend to produce more accurate results. A common distinction between [...]s is that of foxes vs. hedgehogs, based on Isiah Berlin's famous essay.
Contents
1 Foxes and Hedgehogs
2 Blog posts
3 See also
4 Other resources
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Foxes and Hedgehogs
This common distinction comes primarily from the essay "The Hedgehog and the Fox" by Isiah Berlin, regarding the Russian author Leo Tolstoy's theory of history.
The title is a reference to a fragment attributed to the ancient Greek poet Archilochus: πόλλ' οἶδ' ἀλώπηξ, ἐχῖνος δ'ἓν μέγα ("The fox knows many things, but the hedgehog knows one big thing").
Berlin expands upon this idea to divide writers and thinkers into two categories: hedgehogs, who view the world through the lens of a single defining idea and foxes who draw on a wide variety of experiences and for whom the world cannot be boiled down to a single idea.
This distinction is used by Philip Tetlock in the book Expert Political Judgement, which concludes that foxes tend to be better calibrated and more accurate[1].
A: Cognitive style

Q: Coherent Aggregated Volition
A: Coherent Aggregated Volition is Ben Goertzel's response to Eliezer Yudkowsky's Coherent Extrapolated Volition. CAV would be a combination of the goals and beliefs of humanity at the present time. Without the "extrapolation" aspect of CEV, Coherent Aggregated Volition is simpler, and intended to be easier to formalize and prototype in the foreseeable future. CAV is not intended to answer the question of Friendly AI.

Q: [...] is Ben Goertzel's response to Eliezer Yudkowsky's Coherent Extrapolated Volition. CAV would be a combination of the goals and beliefs of humanity at the present time. Without the "extrapolation" aspect of CEV, [...] is simpler, and intended to be easier to formalize and prototype in the foreseeable future. CAV is not intended to answer the question of Friendly AI.
A: Coherent Aggregated Volition

Q: Coherent Extrapolated Volition
A: On the topic of Friendly AI, Eliezer Yudkowsky argues that it would not be sufficient to explicitly program our desires into an AI. Rather, we must  To truly act in our best interests, a superintelligent AI must do what we want it to do, rather than what we tell it to do. Therefore, what we need is a meta-level method of determining what the object-level goal will be. Yudkowsky has presented Coherent Extrapolated Volition as an example of what the solution to this problem would look like.
In calculating CEV, an AI would predict what an idealized version of us would want, "if we knew more, thought faster, were more the people we wished we were, had grown up farther together". It would recursively iterate this prediction for humanity as a whole, and determine the desires which converge. This initial dynamic would be used to form the AI's utility function.
Problems with CEV include the great difficulty of implementing such a program, and the potential that human values may not converge. Yudkowsky considered CEV obsolete almost immediately after its publication in 2004. Despite this, criticism of CEV has been a popular topic of criticism on LessWrong.

Q: On the topic of Friendly AI, Eliezer Yudkowsky argues that it would not be sufficient to explicitly program our desires into an AI. Rather, we must  To truly act in our best interests, a superintelligent AI must do what we want it to do, rather than what we tell it to do. Therefore, what we need is a meta-level method of determining what the object-level goal will be. Yudkowsky has presented [...] as an example of what the solution to this problem would look like.
In calculating CEV, an AI would predict what an idealized version of us would want, "if we knew more, thought faster, were more the people we wished we were, had grown up farther together". It would recursively iterate this prediction for humanity as a whole, and determine the desires which converge. This initial dynamic would be used to form the AI's utility function.
Problems with CEV include the great difficulty of implementing such a program, and the potential that human values may not converge. Yudkowsky considered CEV obsolete almost immediately after its publication in 2004. Despite this, criticism of CEV has been a popular topic of criticism on LessWrong.
A: Coherent Extrapolated Volition

Q: Common knowledge
A: In the context of Aumann's agreement theorem, a fact is part of the common knowledge of a group of agents when they all know it, they all know that they all know it, and so on ad infinitum.

Q: In the context of Aumann's agreement theorem, a fact is part of the [...] of a group of agents when they all know it, they all know that they all know it, and so on ad infinitum.
A: Common knowledge

Q: Common priors
A: A precondition of Aumann's agreement theorem and its numerous generalizations is that the agents must have the same priors. Without common priors, the agents can disagree easily. One can justify the assumption of common priors by noting that it would be awfully strange if rational beliefs could depend on seemingly arbitrary starting features of the agent.
External References
Uncommon Priors Require Origin Disputes (PDF) by Robin Hanson

Q: A precondition of Aumann's agreement theorem and its numerous generalizations is that the agents must have the same priors. Without [...], the agents can disagree easily. One can justify the assumption of [...] by noting that it would be awfully strange if rational beliefs could depend on seemingly arbitrary starting features of the agent.
External References
Uncommon Priors Require Origin Disputes (PDF) by Robin Hanson
A: Common priors

Q: Compartmentalization
A: Compartmentalization, or division of types of activity on separate magisteria is a tendency to restrict application of a generally-applicable skill, such as scientific method, only to select few contexts. More generally, the concept refers to not following a piece of knowledge to its logical conclusion, or not taking it seriously.
A common phenomenon in humans is excellent epistemological performance in one domain - and terrible performance in others. This phenomenon undermines 'appeals to authority' but also shows that someone holding an obviously false belief can still be worth learning from.

Q: [...], or division of types of activity on separate magisteria is a tendency to restrict application of a generally-applicable skill, such as scientific method, only to select few contexts. More generally, the concept refers to not following a piece of knowledge to its logical conclusion, or not taking it seriously.
A common phenomenon in humans is excellent epistemological performance in one domain - and terrible performance in others. This phenomenon undermines 'appeals to authority' but also shows that someone holding an obviously false belief can still be worth learning from.
A: Compartmentalization

Q: Complexity of value
A: Complexity of value is the thesis that human values have high Kolmogorov complexity; that our preferences, the things we care about, cannot be summed by a few simple rules, or compressed.
Many human choices can be compressed, by representing them by simple rules - the desire to survive produces innumerable actions and subgoals as we fulfil that desire.  But people don't just want to survive - although you can compress many human activities to that desire, you cannot compress all of human existence into it.  The human equivalents of a utility function, our terminal values, contain many different elements that are not strictly reducible to one another.  William Frankena offered this list of things which many cultures and people seem to value (for their own sake rather than strictly for their external consequences):
"Life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment, etc.; truth; knowledge and true opinions of various kinds, understanding, wisdom; beauty, harmony, proportion in objects contemplated; aesthetic experience; morally good dispositions or virtues; mutual affection, love, friendship, cooperation; just distribution of goods and evils; harmony and proportion in one's own life; power and experiences of achievement; self-expression; freedom; peace, security; adventure and novelty; and good reputation, honor, esteem, etc."
The complexity of value is a major theme of Eliezer Yudkowsky's writing for two reasons:

Q: [...] is the thesis that human values have high Kolmogorov complexity; that our preferences, the things we care about, cannot be summed by a few simple rules, or compressed.
Many human choices can be compressed, by representing them by simple rules - the desire to survive produces innumerable actions and subgoals as we fulfil that desire.  But people don't just want to survive - although you can compress many human activities to that desire, you cannot compress all of human existence into it.  The human equivalents of a utility function, our terminal values, contain many different elements that are not strictly reducible to one another.  William Frankena offered this list of things which many cultures and people seem to value (for their own sake rather than strictly for their external consequences):
"Life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment, etc.; truth; knowledge and true opinions of various kinds, understanding, wisdom; beauty, harmony, proportion in objects contemplated; aesthetic experience; morally good dispositions or virtues; mutual affection, love, friendship, cooperation; just distribution of goods and evils; harmony and proportion in one's own life; power and experiences of achievement; self-expression; freedom; peace, security; adventure and novelty; and good reputation, honor, esteem, etc."
The [...] is a major theme of Eliezer Yudkowsky's writing for two reasons:
A: Complexity of value

Q: Conceptual metaphor
A: In contemporary linguistics and cognitive science, conceptual metaphors are neurally-implemented mappings between concrete domains of discourse (often related to our body and perception) and more abstract domains. These are a well-known source of bias and are often exploited in the Dark Arts.
The conceptual metaphor "ARGUMENT is WAR" is frequently discussed at Less Wrong, since it is at the core of the Western adversarial system of debate.  It is reflected in such statements as "I won that argument", "His position is indefensible", "He shot down my claim", "His criticisms were right on target" and the like.
Similarly, George Lakoff has documented the effect of political metaphors, which tend to map government to a "nurturing mother" or an "authoritarian father".

Q: In contemporary linguistics and cognitive science, [...]s are neurally-implemented mappings between concrete domains of discourse (often related to our body and perception) and more abstract domains. These are a well-known source of bias and are often exploited in the Dark Arts.
The [...] "ARGUMENT is WAR" is frequently discussed at Less Wrong, since it is at the core of the Western adversarial system of debate.  It is reflected in such statements as "I won that argument", "His position is indefensible", "He shot down my claim", "His criticisms were right on target" and the like.
Similarly, George Lakoff has documented the effect of political metaphors, which tend to map government to a "nurturing mother" or an "authoritarian father".
A: Conceptual metaphor

Q: Configuration space
A: A configuration space is an isomorphism between the attributes of something, and its position on a multidimensional graph. Theoretically, the attributes and precise position on the graph should contain the same information. In practice, the concept usually appears as a suffix, as in "walletspace", where "walletspace" refers to the configuration space of all possible wallets, arranged by similarity. Walletspace would intersect with leatherspace, and the set of leather wallets is a subset of both walletspace and leatherspace, which are both subsets of thingspace.

Q: A [...] is an isomorphism between the attributes of something, and its position on a multidimensional graph. Theoretically, the attributes and precise position on the graph should contain the same information. In practice, the concept usually appears as a suffix, as in "walletspace", where "walletspace" refers to the [...] of all possible wallets, arranged by similarity. Walletspace would intersect with leatherspace, and the set of leather wallets is a subset of both walletspace and leatherspace, which are both subsets of thingspace.
A: Configuration space

Q: Conformity bias
A: Conformity bias is a tendency to behave similarly to the others in a group, even if doing so goes against your own judgment.

Q: [...] is a tendency to behave similarly to the others in a group, even if doing so goes against your own judgment.
A: Conformity bias

Q: Conjunction fallacy
A: The conjunction fallacy consists in assuming that specific conditions are more probable than more general ones.
For the reasons related to representativeness heuristic, a fleshed-out story that contains typical amount of detail sounds more plausible than a stripped-down description of a situation that only states a few facts. There is a tendency for people to take that plausibility at face value, and assign probability accordingly. This intuition is wrong, because the conjunction rule of probability theory states that, for any event X, its conjunction with additional details Y will be less probable.
The conjunction fallacy suggests that one should be very careful in adding details to any claim, as even though each such detail may make the claim so much more convincing, it also inevitably subtracts from its validity.

Q: The [...] consists in assuming that specific conditions are more probable than more general ones.
For the reasons related to representativeness heuristic, a fleshed-out story that contains typical amount of detail sounds more plausible than a stripped-down description of a situation that only states a few facts. There is a tendency for people to take that plausibility at face value, and assign probability accordingly. This intuition is wrong, because the conjunction rule of probability theory states that, for any event X, its conjunction with additional details Y will be less probable.
The [...] suggests that one should be very careful in adding details to any claim, as even though each such detail may make the claim so much more convincing, it also inevitably subtracts from its validity.
A: Conjunction fallacy

Q: Connotation
A: Connotation is a subjective cultural and/or emotional coloration in addition to the explicit or denotative meaning of any specific word or phrase in a language, i.e. emotional association with a word.

Q: [...] is a subjective cultural and/or emotional coloration in addition to the explicit or denotative meaning of any specific word or phrase in a language, i.e. emotional association with a word.
A: Connotation

Q: Consequentialism
A: Consequentialism is an array of ethical theories which hold that the goodness or rightness of an action is to be measured by the sum of all its outcomes. It is one of the two most major strands of contemporary ethics. The other, deontology, measures the morality of an action by its compliance to given rules and moral precepts, that must be followed above all else.
The consequentialist analysis can take the form of a utility function, where the expected utility of an action is determined by the sum of the utility of each of its possible consequences, individually weighted by their respective probability of occurrence. Therefore, the correct action is always the one which maximizes o total value of positive consequences, even if it violates some established moral rule. Consequentialism is based on a cost and benefits analysis and measure goodness by calculating the total expected good. In this manner, opposing to deontological theories, there is no action good in itself or bad in itself, they have to be judged by its results.
Which outcomes are to be considered good or bad is an open question to consequentialism. Hence, one can egoistically value his own well fare in detriment of others and still act according to a consequentialist ethics. Utilitarianism can be seen as instance of consequentialism where one must aim at maximizing the overall good, the happiness and welfare of himself and others.
One common tool for detecting a person way of thinking about morality is the use of moral dilemmas, in special the trolley’s problems. Those problems have generated a vast array of moral dilemmas of hard resolution and justification. They usually follow a general schema:  “Person A can take an action which will benefit a great number of people, but that also violates moral rule X or person’s B individual rights. What should person A do?” Taking the action reveals a consequentialist thinking, whereas holding to the moral rule reveals a deontological thinking.

Q: [...] is an array of ethical theories which hold that the goodness or rightness of an action is to be measured by the sum of all its outcomes. It is one of the two most major strands of contemporary ethics. The other, deontology, measures the morality of an action by its compliance to given rules and moral precepts, that must be followed above all else.
The consequentialist analysis can take the form of a utility function, where the expected utility of an action is determined by the sum of the utility of each of its possible consequences, individually weighted by their respective probability of occurrence. Therefore, the correct action is always the one which maximizes o total value of positive consequences, even if it violates some established moral rule. [...] is based on a cost and benefits analysis and measure goodness by calculating the total expected good. In this manner, opposing to deontological theories, there is no action good in itself or bad in itself, they have to be judged by its results.
Which outcomes are to be considered good or bad is an open question to [...]. Hence, one can egoistically value his own well fare in detriment of others and still act according to a consequentialist ethics. Utilitarianism can be seen as instance of [...] where one must aim at maximizing the overall good, the happiness and welfare of himself and others.
One common tool for detecting a person way of thinking about morality is the use of moral dilemmas, in special the trolley’s problems. Those problems have generated a vast array of moral dilemmas of hard resolution and justification. They usually follow a general schema:  “Person A can take an action which will benefit a great number of people, but that also violates moral rule X or person’s B individual rights. What should person A do?” Taking the action reveals a consequentialist thinking, whereas holding to the moral rule reveals a deontological thinking.
A: Consequentialism

Q: Conservation of expected evidence
A: Conservation of expected evidence is a theorem that says: "for every expectation of evidence, there is an equal and opposite expectation of counterevidence".
Consider a hypothesis H and evidence (observation) E. Prior probability of the hypothesis is P(H); posterior probability is either P(H|E) or P(H|¬E), depending on whether you observe E or not-E (evidence or counterevidence). The probability of observing E is P(E), and probability of observing not-E is P(¬E). Thus, expected value of the posterior probability of the hypothesis is:
But the prior probability of the hypothesis itself can be trivially broken up the same way:
Thus, expectation of posterior probability is equal to the prior probability.
In other way, if you expect the probability of a hypothesis to change as a result of observing some evidence, the amount of this change if the evidence is positive is:
If the evidence is negative, the change is:
Expectation of the change given positive evidence is equal to negated expectation of the change given counterevidence:
If you can anticipate in advance updating your belief in a particular direction, then you should just go ahead and update now. Once you know your destination, you are already there. On pain of paradox, a low probability of seeing strong evidence in one direction must be balanced by a high probability of observing weak counterevidence in the other direction.

Q: [...] is a theorem that says: "for every expectation of evidence, there is an equal and opposite expectation of counterevidence".
Consider a hypothesis H and evidence (observation) E. Prior probability of the hypothesis is P(H); posterior probability is either P(H|E) or P(H|¬E), depending on whether you observe E or not-E (evidence or counterevidence). The probability of observing E is P(E), and probability of observing not-E is P(¬E). Thus, expected value of the posterior probability of the hypothesis is:
But the prior probability of the hypothesis itself can be trivially broken up the same way:
Thus, expectation of posterior probability is equal to the prior probability.
In other way, if you expect the probability of a hypothesis to change as a result of observing some evidence, the amount of this change if the evidence is positive is:
If the evidence is negative, the change is:
Expectation of the change given positive evidence is equal to negated expectation of the change given counterevidence:
If you can anticipate in advance updating your belief in a particular direction, then you should just go ahead and update now. Once you know your destination, you are already there. On pain of paradox, a low probability of seeing strong evidence in one direction must be balanced by a high probability of observing weak counterevidence in the other direction.
A: Conservation of expected evidence

Q: Contagion heuristic
A: The contagion heuristic is a psychological heuristic leading people to avoid contact with people or objects viewed as "contaminated" by previous contact with someone or something viewed as bad — or, less often, to seek contact with objects that have been in contact with people or things considered good.

Q: The [...] is a psychological heuristic leading people to avoid contact with people or objects viewed as "contaminated" by previous contact with someone or something viewed as bad — or, less often, to seek contact with objects that have been in contact with people or things considered good.
A: Contagion heuristic

Q: Control theory
A: A control system is a device that keeps a variable at a certain value, despite only knowing what the current value of the variable is.  An example is a cruise control, which maintains a certain speed, but only measures the current speed, and knows nothing of the system that produces that speed (wind, car weight, grade).

Q: A control system is a device that keeps a variable at a certain value, despite only knowing what the current value of the variable is.  An example is a cruise control, which maintains a certain speed, but only measures the current speed, and knows nothing of the system that produces that speed (wind, car weight, grade).
A: Control theory

Q: Correspondence bias
A: Correspondence bias (also known as the fundamental attribution error) is the tendency to overestimate the the contribution of lasting traits and dispositions in determining people's behavior, as compared to situational effects. We might see someone kicking a vending machine, and conclude they're an inherently angry person. But maybe they just failed a test, had their driving license revoked, and had the machine eat their money for the third time this week. We think of the other person as an evil mutant and ourselves as righteous actors.

Q: [...] (also known as the fundamental attribution error) is the tendency to overestimate the the contribution of lasting traits and dispositions in determining people's behavior, as compared to situational effects. We might see someone kicking a vending machine, and conclude they're an inherently angry person. But maybe they just failed a test, had their driving license revoked, and had the machine eat their money for the third time this week. We think of the other person as an evil mutant and ourselves as righteous actors.
A: Correspondence bias

Q: Corrupted hardware
A: Our brains do not always allow us to act the way we should. Corrupted hardware refers to those behaviors and thoughts that act for ancestrally relevant purposes rather than for stated moralities and preferences.

Q: Our brains do not always allow us to act the way we should. [...] refers to those behaviors and thoughts that act for ancestrally relevant purposes rather than for stated moralities and preferences.
A: Corrupted hardware

Q: Costs of rationality
A: Becoming more epistemically rational can only guarantee one thing: what you believe will include more of the truth. Knowing that truth might help you achieve your goals, or cause you to become a pariah. Be sure that you really want to know the truth before you commit to finding it; otherwise, you may flinch from it.

Q: Becoming more epistemically rational can only guarantee one thing: what you believe will include more of the truth. Knowing that truth might help you achieve your goals, or cause you to become a pariah. Be sure that you really want to know the truth before you commit to finding it; otherwise, you may flinch from it.
A: Costs of rationality

Q: Counterfactual mugging
A: Counterfactual mugging is a thought experiment for testing decision theories stated as follows:
Omega appears and says that it has just tossed a fair coin, and given that the coin came up tails, it decided to ask you to give it $100. Whatever you do in this situation, nothing else will happen differently in reality as a result. Naturally you don't want to give up your $100. But Omega also tells you that if the coin came up heads instead of tails, it'd give you $10000, but only if you'd agree to give it $100 if the coin came up tails.

Q: [...] is a thought experiment for testing decision theories stated as follows:
Omega appears and says that it has just tossed a fair coin, and given that the coin came up tails, it decided to ask you to give it $100. Whatever you do in this situation, nothing else will happen differently in reality as a result. Naturally you don't want to give up your $100. But Omega also tells you that if the coin came up heads instead of tails, it'd give you $10000, but only if you'd agree to give it $100 if the coin came up tails.
A: Counterfactual mugging

Q: Cox\'s theorem
A: Cox's theorem states that any system of representing beliefs that satisfies certain criteria of reasonableness must be structurally equivalent to probability theory.
Along with the Dutch book argument, Cox's theorem is an argument for considering probability theory as a normative theory of reasoning.

Q: [...] states that any system of representing beliefs that satisfies certain criteria of reasonableness must be structurally equivalent to probability theory.
Along with the Dutch book argument, [...] is an argument for considering probability theory as a normative theory of reasoning.
A: Cox\'s theorem

Q: Crisis of faith
A: Crisis of faith is a combined technique for recognizing and eradicating the whole systems of mutually-supporting false beliefs. The technique involves systematic application of introspection, with the express intent to check the reliability of beliefs independently of the other beliefs that support them in the mind. The technique might be useful for the victims of affective death spirals, or any other systematic confusions, especially those supported by anti-epistemology.

Q: [...] is a combined technique for recognizing and eradicating the whole systems of mutually-supporting false beliefs. The technique involves systematic application of introspection, with the express intent to check the reliability of beliefs independently of the other beliefs that support them in the mind. The technique might be useful for the victims of affective death spirals, or any other systematic confusions, especially those supported by anti-epistemology.
A: Crisis of faith

Q: Crocker\'s rules
A: Crocker's rules, named after and framed by Lee Daniel Crocker, is a social communication protocol or etiquette used to reduce emotional impact on debate.  Crocker was an early contributor to Wikipedia, and, with Larry Sanger, Fred Bauder and others who chose to contribute pseudonymously or anonymously, helped to form its rules to maximize objective reporting.
By declaring commitment to Crocker's rules, one authorizes other debaters to optimize their messages for information, even when this entails that emotional feelings will be disregarded. The underlying assumption is that rudeness is sometimes necessary for effective conveyance of information, if only to signal a lack of patience or tolerance: after all, knowing whether the speaker is becoming angry or despondent is useful rational evidence.
Thus, one who has committed to these rules largely gives up the right to complain about emotional provocation, "flaming", "trolling", "abuse" (hopelessly subjective terms) and other alleged violations of etiquette.  They give these rights up in the interest of effective debate.
Contents
1 Similar protocols (contrasted)
2 Blog posts
3 External links
4 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
  Similar protocols (contrasted)
In contrast to radical honesty, Crocker's rules encourage being tactful with anyone who hasn't specifically accepted them. This follows the general principle of being "liberal in what you accept and conservative in what you send".
Crocker emphasized, repeatedly, in Wikipedia discourse and elsewhere, that one could only adopt Crocker's rules to apply to oneself, and could not impose them on a debate or forum with participants who had not opted-in explicitly to these rules, nor use them to exclude any participant.
A lack of freedom to make emotionally uncomfortable observations is often thought to lead to groupthink.  The assertion that unanimity is not consensus arises from common anecdotal evidence that disagreement is often not expressed not because it is not relevant, but not pleasant, and that poor decisions are often unchallenged as a result.  This principle is ancient, and dates back at least to the ancient Hebrew Sanhedrin wherein unanimous decisions were considered unfair by nature and represented only the lack of adequate defense of the accused.  A voluntary adoption of Crocker's rules would presumably eliminate at least some cases of disagreeable silence and make more objections explicit, as in the practice of appeals and supreme courts, where the minority opinion is codified with the same care as is taken with the majority's.

Q: [...], named after and framed by Lee Daniel Crocker, is a social communication protocol or etiquette used to reduce emotional impact on debate.  Crocker was an early contributor to Wikipedia, and, with Larry Sanger, Fred Bauder and others who chose to contribute pseudonymously or anonymously, helped to form its rules to maximize objective reporting.
By declaring commitment to [...], one authorizes other debaters to optimize their messages for information, even when this entails that emotional feelings will be disregarded. The underlying assumption is that rudeness is sometimes necessary for effective conveyance of information, if only to signal a lack of patience or tolerance: after all, knowing whether the speaker is becoming angry or despondent is useful rational evidence.
Thus, one who has committed to these rules largely gives up the right to complain about emotional provocation, "flaming", "trolling", "abuse" (hopelessly subjective terms) and other alleged violations of etiquette.  They give these rights up in the interest of effective debate.
Contents
1 Similar protocols (contrasted)
2 Blog posts
3 External links
4 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
  Similar protocols (contrasted)
In contrast to radical honesty, [...] encourage being tactful with anyone who hasn't specifically accepted them. This follows the general principle of being "liberal in what you accept and conservative in what you send".
Crocker emphasized, repeatedly, in Wikipedia discourse and elsewhere, that one could only adopt [...] to apply to oneself, and could not impose them on a debate or forum with participants who had not opted-in explicitly to these rules, nor use them to exclude any participant.
A lack of freedom to make emotionally uncomfortable observations is often thought to lead to groupthink.  The assertion that unanimity is not consensus arises from common anecdotal evidence that disagreement is often not expressed not because it is not relevant, but not pleasant, and that poor decisions are often unchallenged as a result.  This principle is ancient, and dates back at least to the ancient Hebrew Sanhedrin wherein unanimous decisions were considered unfair by nature and represented only the lack of adequate defense of the accused.  A voluntary adoption of [...] would presumably eliminate at least some cases of disagreeable silence and make more objections explicit, as in the practice of appeals and supreme courts, where the minority opinion is codified with the same care as is taken with the majority's.
A: Crocker\'s rules

Q: Cryonics
A: Cryonics is the practice of preserving people who are dying in liquid nitrogen soon after their heart stops. The idea is that most of your brain's information content is still intact right after you've "died". If humans invent molecular nanotechnology or brain emulation techniques, it may be possible to reconstruct the consciousness of cryopreserved patients.
Contents
1 Cryonics-associated issues commonly raised on Less Wrong
2 Blog posts
3 External links
4 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Cryonics-associated issues commonly raised on Less Wrong
Pro-cryonics points:
Advanced reductionism/physicalism (because of the issues associated with identifying a person with continuity of brain information).
Whether an extended healthy lifespan is worthwhile (relates to Fun Theory, religious rationalizations for 70-year lifespans, "sour grapes" rationalizations for why death is actually a good thing).
The "shut up and multiply" aspect of spending $300/year (as Eliezer Yudkowsky quotes his costs for Cryonics Institute membership ($125/year) plus term life insurance ($180/year)) for a probability (how large being widely disputed) of obtaining many more years of lifespan.  For this reason, cryonics advocates regard it as an extreme case of failure at rationality - a low-hanging fruit by which millions of deaths per year could be prevented at low cost.
Anti-cryonics points:

Q: [...] is the practice of preserving people who are dying in liquid nitrogen soon after their heart stops. The idea is that most of your brain's information content is still intact right after you've "died". If humans invent molecular nanotechnology or brain emulation techniques, it may be possible to reconstruct the consciousness of cryopreserved patients.
Contents
1 [...]-associated issues commonly raised on Less Wrong
2 Blog posts
3 External links
4 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
[...]-associated issues commonly raised on Less Wrong
Pro-[...] points:
Advanced reductionism/physicalism (because of the issues associated with identifying a person with continuity of brain information).
Whether an extended healthy lifespan is worthwhile (relates to Fun Theory, religious rationalizations for 70-year lifespans, "sour grapes" rationalizations for why death is actually a good thing).
The "shut up and multiply" aspect of spending $300/year (as Eliezer Yudkowsky quotes his costs for [...] Institute membership ($125/year) plus term life insurance ($180/year)) for a probability (how large being widely disputed) of obtaining many more years of lifespan.  For this reason, [...] advocates regard it as an extreme case of failure at rationality - a low-hanging fruit by which millions of deaths per year could be prevented at low cost.
Anti-[...] points:
A: Cryonics

Q: Curiosity
A: The first virtue is curiosity. A burning itch to know is higher than a solemn vow to pursue truth. To feel the burning itch of curiosity requires both that you be ignorant, and that you desire to relinquish your ignorance. If in your heart you believe you already know, or if in your heart you do not wish to know, then your questioning will be purposeless and your skills without direction. Curiosity seeks to annihilate itself; there is no curiosity that does not want an answer. The glory of glorious mystery is to be solved, after which it ceases to be mystery. Be wary of those who speak of being open-minded and modestly confess their ignorance. There is a time to confess your ignorance and a time to relinquish your ignorance.

Q: The first virtue is [...]. A burning itch to know is higher than a solemn vow to pursue truth. To feel the burning itch of [...] requires both that you be ignorant, and that you desire to relinquish your ignorance. If in your heart you believe you already know, or if in your heart you do not wish to know, then your questioning will be purposeless and your skills without direction. [...] seeks to annihilate itself; there is no [...] that does not want an answer. The glory of glorious mystery is to be solved, after which it ceases to be mystery. Be wary of those who speak of being open-minded and modestly confess their ignorance. There is a time to confess your ignorance and a time to relinquish your ignorance.
A: Curiosity

Q: Semantic stopsign
A: Semantic stopsign or curiosity stopper is a meaningless generic explanation that creates an illusion of giving an answer, without actually explaining anything. Semantic stopsigns destroy curiosity, giving surrogate answers and stopping the search for truth prematurely. Can preserve incorrect beliefs for a long time, insisting on following cached thought without rethinking anything. A tool of dark arts and important part of any anti-epistemology.
When specific words act as stopsigns, placing a rationalist taboo on them may help. Epistemic hygiene allows to exclude some of the stopsigns raised by partial arguers or your own cognitive biases.

Q: [...] or curiosity stopper is a meaningless generic explanation that creates an illusion of giving an answer, without actually explaining anything. [...]s destroy curiosity, giving surrogate answers and stopping the search for truth prematurely. Can preserve incorrect beliefs for a long time, insisting on following cached thought without rethinking anything. A tool of dark arts and important part of any anti-epistemology.
When specific words act as stopsigns, placing a rationalist taboo on them may help. Epistemic hygiene allows to exclude some of the stopsigns raised by partial arguers or your own cognitive biases.
A: Semantic stopsign

Q: Dangerous knowledge
A: Intelligence, in order to be useful, must be used for something other than defeating itself. Knowledge of human heuristics and biases may cause people to selectively find those biases in arguments they disagree with. This ability to destroy arguments that aren't already accepted will make a more knowledgeable person less able to change their views when presented with evidence.
Even simple facts, when learned superficially, can turn against you, for reasons related to representativeness bias. Norbert Schwarz et al. write[1]:
Public
information campaigns that confront myths with facts, or warn people that a
given claim is false, necessarily reiterate the information they want to discredit. [...]
When the false claims are encountered again on a later occasion, all that is left may be the vague feeling that "I heard something like this before." This sense of familiarity, in turn, will foster the acceptance of statements as true.

Q: Intelligence, in order to be useful, must be used for something other than defeating itself. Knowledge of human heuristics and biases may cause people to selectively find those biases in arguments they disagree with. This ability to destroy arguments that aren't already accepted will make a more knowledgeable person less able to change their views when presented with evidence.
Even simple facts, when learned superficially, can turn against you, for reasons related to representativeness bias. Norbert Schwarz et al. write[1]:
Public
information campaigns that confront myths with facts, or warn people that a
given claim is false, necessarily reiterate the information they want to discredit. [...]
When the false claims are encountered again on a later occasion, all that is left may be the vague feeling that "I heard something like this before." This sense of familiarity, in turn, will foster the acceptance of statements as true.
A: Dangerous knowledge

Q: Dark arts
A: The term Dark Arts refers to rhetorical techniques crafted to exploit human cognitive biases in order to persuade, deceive, or otherwise manipulate a person into irrationally accepting beliefs perpetuated by the practitioner of the Arts.  Use of the dark arts is especially common in sales and similar situations (known as hard sell in the sales business) and promotion of political and religious views.
Contents
1 Art vs. tech
2 Blog posts
3 Other links
4 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
  Art vs. tech
Sometimes these arts are further augmented by the use of persuasion technology, such as broadcast advertising or PowerPoint slides.  Persuasion technology may prevent the person who is being targeted from carefully deliberating on the intended message, or thinking up an effective response to it in real time.
Such effects can be caused by something as benign as the use of a specialist vocabulary which the target is unfamiliar with, or an institutional vocabulary with high-status connotations: this is one reason why many specialist professions employ ethical codes to regulate their unbalanced power relationship with customers.
The use of such techniques as whiteboards or PowerPoint slides brings additional concerns, since these tend to connote a single party as the one "in charge" of the presentation: this makes it even more difficult for the intended audience to raise any effective objection, and encourages them to focus their attention on the content of the whiteboard or slides.  Said content is often presented as a list of abrupt "bullet points", further connoting it as factual, objective and neutral.  One outspoken critic of PowerPoint, management professor David R. Beatty, states: "It is like a disease. It's the AIDS of management."  Beatty further states that Powerpoint "removes subtlety and thinking".
Many futurists expect that a technological singularity of even a very mild character will lead to an explosion in the use of radically effective persuasive technology, or "cognotechnology"--a term coined by American military researchers at the Lawrence Livermore Laboratories.  The collection and distribution of information about people may spiral out beyond any feasible control, perhaps even comprising their inner thought processes; cognitive monitoring may range from non-intrusive body monitoring as seen in a polygraph to outright brain emulation.  In this scenario, persuasion technology may easily blend over into outright mind control.  This is clearly a rather paranoiac and dystopian scenario; nevertheless, the fact that it is being seriously discussed has persuasive potential in itself, such as for directing funding for research into guaranteed Friendly AI, as opposed to naïvely pursuing expanded funding for neuroscience or artificial intelligence.

Q: The term Dark Arts refers to rhetorical techniques crafted to exploit human cognitive biases in order to persuade, deceive, or otherwise manipulate a person into irrationally accepting beliefs perpetuated by the practitioner of the Arts.  Use of the [...] is especially common in sales and similar situations (known as hard sell in the sales business) and promotion of political and religious views.
Contents
1 Art vs. tech
2 Blog posts
3 Other links
4 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
  Art vs. tech
Sometimes these arts are further augmented by the use of persuasion technology, such as broadcast advertising or PowerPoint slides.  Persuasion technology may prevent the person who is being targeted from carefully deliberating on the intended message, or thinking up an effective response to it in real time.
Such effects can be caused by something as benign as the use of a specialist vocabulary which the target is unfamiliar with, or an institutional vocabulary with high-status connotations: this is one reason why many specialist professions employ ethical codes to regulate their unbalanced power relationship with customers.
The use of such techniques as whiteboards or PowerPoint slides brings additional concerns, since these tend to connote a single party as the one "in charge" of the presentation: this makes it even more difficult for the intended audience to raise any effective objection, and encourages them to focus their attention on the content of the whiteboard or slides.  Said content is often presented as a list of abrupt "bullet points", further connoting it as factual, objective and neutral.  One outspoken critic of PowerPoint, management professor David R. Beatty, states: "It is like a disease. It's the AIDS of management."  Beatty further states that Powerpoint "removes subtlety and thinking".
Many futurists expect that a technological singularity of even a very mild character will lead to an explosion in the use of radically effective persuasive technology, or "cognotechnology"--a term coined by American military researchers at the Lawrence Livermore Laboratories.  The collection and distribution of information about people may spiral out beyond any feasible control, perhaps even comprising their inner thought processes; cognitive monitoring may range from non-intrusive body monitoring as seen in a polygraph to outright brain emulation.  In this scenario, persuasion technology may easily blend over into outright mind control.  This is clearly a rather paranoiac and dystopian scenario; nevertheless, the fact that it is being seriously discussed has persuasive potential in itself, such as for directing funding for research into guaranteed Friendly AI, as opposed to naïvely pursuing expanded funding for neuroscience or artificial intelligence.
A: Dark arts

Q: Death
A: First you're there, and then you're not there, and they can't change you from being not there to being there, because there's nothing there to be changed from being not there to being there. That's death.
Cryonicists use the concept of information-theoretic death, which is what happens when the information needed to reconstruct you even in principle is no longer present. Anything less, to them, is just a flesh wound.

Q: First you're there, and then you're not there, and they can't change you from being not there to being there, because there's nothing there to be changed from being not there to being there. That's [...].
Cryonicists use the concept of information-theoretic [...], which is what happens when the information needed to reconstruct you even in principle is no longer present. Anything less, to them, is just a flesh wound.
A: Death

Q: Death Spirals and the Cult Attractor
A: A subsequence of How to Actually Change Your Mind on two of the huger obstacles, the affective death spiral and the cultishness attractor.
Affective death spirals are positive feedback loop caused by the halo effect:  Positive characteristics perceptually correlate, so the more nice things we say about X, the more additional nice things we're likely to believe about X.
Cultishness is an empirical attractor in human groups, roughly an affective death spiral, plus peer pressure and outcasting behavior, plus (quite often) defensiveness around something believed to have been perfected.
Some overlap with the Politics is the Mind-Killer subsequence (preferably read that first).

Q: A subsequence of How to Actually Change Your Mind on two of the huger obstacles, the affective death spiral and the cultishness attractor.
Affective death spirals are positive feedback loop caused by the halo effect:  Positive characteristics perceptually correlate, so the more nice things we say about X, the more additional nice things we're likely to believe about X.
Cultishness is an empirical attractor in human groups, roughly an affective death spiral, plus peer pressure and outcasting behavior, plus (quite often) defensiveness around something believed to have been perfected.
Some overlap with the Politics is the Mind-Killer subsequence (preferably read that first).
A: Death Spirals and the Cult Attractor

Q: Debiasing
A: The process of overcoming bias. Results in being able to systematically avoid stupidity in certain situations where intuitive heuristics run amok. It takes serious study to gain meaningful benefits, half-hearted attempts may accomplish nothing, and partial knowledge of bias may do more harm than good.
Simply being told about the biases rarely helps to notice and overcome them. For example, the overestimation of prior probability of an event that has already happened caused by hindsight bias doesn't go away when people are told about this bias and when they try to counteract its effects. Due to overconfidence, often more than 50% of people report that they are capable of above-average performance. When asked about how good they are at overcoming overconfidence, they exhibit the same bias[1].

Q: The process of overcoming bias. Results in being able to systematically avoid stupidity in certain situations where intuitive heuristics run amok. It takes serious study to gain meaningful benefits, half-hearted attempts may accomplish nothing, and partial knowledge of bias may do more harm than good.
Simply being told about the biases rarely helps to notice and overcome them. For example, the overestimation of prior probability of an event that has already happened caused by hindsight bias doesn't go away when people are told about this bias and when they try to counteract its effects. Due to overconfidence, often more than 50% of people report that they are capable of above-average performance. When asked about how good they are at overcoming overconfidence, they exhibit the same bias[1].
A: Debiasing

Q: Decision theory
A: Decision theory is the study of principles and algorithms for making correct decisions—that is, decisions that allow an agent to achieve better outcomes with respect to its goals. Every action at least implicitly represents a decision under uncertainty: in a state of partial knowledge, something has to be done, even if that something turns out to be nothing (call it "the null action"). Even if you don't know how you make decisions, decisions do get made, and so there has to be some underlying mechanism. What is it? And how can it be done better? Decision theory has the answers.
A core idea in decision theory is that of expected utility maximization, usually intractable to directly calculate in practice, but an invaluable theoretical concept. An agent assigns utility to every possible outcome: a real number representing the goodness or desirability of that outcome. The mapping of outcomes to utilities is called the agent's utility function. (The utility function is said to be invariant under affine transformations: that is, the utilities can be scaled or translated by a constant while resulting in all the same decisions.) For every action that the agent could take, sum over the utilities of the various possible outcomes weighted by their probability: this is the expected utility of the action, and the action with the highest expected utility is to be chosen.
Contents
1 Thought experiments
2 Blog posts
3 Sequence by AnnaSalamon
4 Sequence by orthonormal (Decision Theories: A Semi-Formal Analysis)
5 Commonly discussed decision theories
6 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Thought experiments
The limitations and pathologies of decision theories can be analyzed by considering the decisions they suggest in the certain idealized situations that stretch the limits of decision theory's applicability. Some of the thought experiments more frequently discussed on LW include:
Newcomb's problem
Counterfactual mugging
Parfit's hitchhiker
Smoker's lesion
Absentminded driver
Sleeping Beauty problem
Prisoner's dilemma
Pascal's mugging
Blog posts
Terminal Values and Instrumental Values
Decision Theories: A Less Wrong Primer by orthonormal
Sequence by AnnaSalamon
Decision theory: An outline of some upcoming posts
Confusion about Newcomb is confusion about counterfactuals
Why we need to reduce “could”, “would”, “should”
Why Pearl helps reduce “could” and “would”, but still leaves us with at least three alternatives
Sequence by orthonormal (Decision Theories: A Semi-Formal Analysis)
Part 0: Decision Theories: A Less Wrong Primer
Part I: The Problem with Naive Decision Theory
Part II: Causal Decision Theory and Substitution
Part III: Formalizing Timeless Decision Theory
Commonly discussed decision theories
Standard theories well-known in academia:
CDT, Causal Decision Theory
EDT, Evidential Decision Theory
Theories invented by researchers associated with SingInst and LW:

Q: [...] is the study of principles and algorithms for making correct decisions—that is, decisions that allow an agent to achieve better outcomes with respect to its goals. Every action at least implicitly represents a decision under uncertainty: in a state of partial knowledge, something has to be done, even if that something turns out to be nothing (call it "the null action"). Even if you don't know how you make decisions, decisions do get made, and so there has to be some underlying mechanism. What is it? And how can it be done better? [...] has the answers.
A core idea in [...] is that of expected utility maximization, usually intractable to directly calculate in practice, but an invaluable theoretical concept. An agent assigns utility to every possible outcome: a real number representing the goodness or desirability of that outcome. The mapping of outcomes to utilities is called the agent's utility function. (The utility function is said to be invariant under affine transformations: that is, the utilities can be scaled or translated by a constant while resulting in all the same decisions.) For every action that the agent could take, sum over the utilities of the various possible outcomes weighted by their probability: this is the expected utility of the action, and the action with the highest expected utility is to be chosen.
Contents
1 Thought experiments
2 Blog posts
3 Sequence by AnnaSalamon
4 Sequence by orthonormal (Decision Theories: A Semi-Formal Analysis)
5 Commonly discussed decision theories
6 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Thought experiments
The limitations and pathologies of decision theories can be analyzed by considering the decisions they suggest in the certain idealized situations that stretch the limits of [...]'s applicability. Some of the thought experiments more frequently discussed on LW include:
Newcomb's problem
Counterfactual mugging
Parfit's hitchhiker
Smoker's lesion
Absentminded driver
Sleeping Beauty problem
Prisoner's dilemma
Pascal's mugging
Blog posts
Terminal Values and Instrumental Values
Decision Theories: A Less Wrong Primer by orthonormal
Sequence by AnnaSalamon
[...]: An outline of some upcoming posts
Confusion about Newcomb is confusion about counterfactuals
Why we need to reduce “could”, “would”, “should”
Why Pearl helps reduce “could” and “would”, but still leaves us with at least three alternatives
Sequence by orthonormal (Decision Theories: A Semi-Formal Analysis)
Part 0: Decision Theories: A Less Wrong Primer
Part I: The Problem with Naive Decision Theory
Part II: Causal Decision Theory and Substitution
Part III: Formalizing Timeless Decision Theory
Commonly discussed decision theories
Standard theories well-known in academia:
CDT, Causal Decision Theory
EDT, Evidential Decision Theory
Theories invented by researchers associated with SingInst and LW:
A: Decision theory

Q: Decoherence
A: Decoherence happens when some quantum system interacts irreversibly with its environment, and different components of the wavefunction stop interfering. In the many-worlds interpretation, this corresponds to the splitting of worlds.

Q: [...] happens when some quantum system interacts irreversibly with its environment, and different components of the wavefunction stop interfering. In the many-worlds interpretation, this corresponds to the splitting of worlds.
A: Decoherence

Q: Defensibility
A: Beware when you find yourself arguing that a policy is defensible rather than optimal; or that it has some benefit compared to the null action, rather than the best benefit of any action.

Q: Beware when you find yourself arguing that a policy is defensible rather than optimal; or that it has some benefit compared to the null action, rather than the best benefit of any action.
A: Defensibility

Q: Defying the data
A: Sometimes, the results of an experiment contradict what we have strong theoretical reason to believe. But experiments can go wrong, for various reasons. So if our theory is strong enough, we should in some cases defy the data: know that there has to be something wrong with the result, even without offering ideas on what it might be.

Q: Sometimes, the results of an experiment contradict what we have strong theoretical reason to believe. But experiments can go wrong, for various reasons. So if our theory is strong enough, we should in some cases defy the data: know that there has to be something wrong with the result, even without offering ideas on what it might be.
A: Defying the data

Q: Detached lever fallacy
A: It is a truism in evolutionary biology that conditional responses require more genetic complexity than unconditional responses.  To develop a fur coat in response to cold weather requires more genetic complexity than developing a fur coat whether or not there is cold weather, because in the former case you also have to develop cold-weather sensors and wire them up to the fur coat.
But this can lead to Lamarckian delusions:  Look, I put the organism in a cold environment, and poof, it develops a fur coat!  Genes?  What genes?  It's the cold that does it, obviously.
This fallacy underlies a form of anthropomorphism in which people expect that, as a universal rule, particular stimuli applied to any mind-in-general will produce some particular response - for example, that if you punch an AI in the nose, it will get angry.  Humans are programmed with that particular conditional response, but not all possible minds would be.
Similarly:  You've seen apples, touched apples, picked them up and held them, bought them for money, cut them into slices, eaten the slices and tasted them.  Your eyes can see the raw pixels of an apple, and your visual cortex process it into a 3D shape, recognized by your temporal lobe as an apple similar (despite differences of shape and color and angle of vision) to other apples in your experience.  Your motor cortex and cerebellum can move your fingers to pick up the apple.
That is all the complex machinery your brain has for apples; and you can pull the lever on that complex machinery just by saying "apple" to a fellow English-speaking human, since the two of you have both learned to associate the sound "ap-ple" to activate all that complicated machinery.
But if that machinery isn't there - if you're writing "apple" as five ASCII letters inside a so-called AI's so-called knowledge base, when the AI can't recognize apples or do anything with apples - then the word is just a lever detached from its underlying machinery.

Q: It is a truism in evolutionary biology that conditional responses require more genetic complexity than unconditional responses.  To develop a fur coat in response to cold weather requires more genetic complexity than developing a fur coat whether or not there is cold weather, because in the former case you also have to develop cold-weather sensors and wire them up to the fur coat.
But this can lead to Lamarckian delusions:  Look, I put the organism in a cold environment, and poof, it develops a fur coat!  Genes?  What genes?  It's the cold that does it, obviously.
This fallacy underlies a form of anthropomorphism in which people expect that, as a universal rule, particular stimuli applied to any mind-in-general will produce some particular response - for example, that if you punch an AI in the nose, it will get angry.  Humans are programmed with that particular conditional response, but not all possible minds would be.
Similarly:  You've seen apples, touched apples, picked them up and held them, bought them for money, cut them into slices, eaten the slices and tasted them.  Your eyes can see the raw pixels of an apple, and your visual cortex process it into a 3D shape, recognized by your temporal lobe as an apple similar (despite differences of shape and color and angle of vision) to other apples in your experience.  Your motor cortex and cerebellum can move your fingers to pick up the apple.
That is all the complex machinery your brain has for apples; and you can pull the lever on that complex machinery just by saying "apple" to a fellow English-speaking human, since the two of you have both learned to associate the sound "ap-ple" to activate all that complicated machinery.
But if that machinery isn't there - if you're writing "apple" as five ASCII letters inside a so-called AI's so-called knowledge base, when the AI can't recognize apples or do anything with apples - then the word is just a lever detached from its underlying machinery.
A: Detached lever fallacy

Q: Disagreement
A: Aumann's agreement theorem can be informally interpreted as suggesting that if two people are honest seekers of truth, and both believe each other to be honest, then they should update on each other's opinions and quickly reach agreement. The very fact that a person believes something is Rational evidence that that something is true, and so this fact should be taken into account when forming your belief.
Outside of well-functioning prediction markets, Aumann agreement can probably only be approximated by careful deliberative discourse.  Thus, fostering effective deliberation should be seen as a key goal of Less Wrong.

Q: Aumann's agreement theorem can be informally interpreted as suggesting that if two people are honest seekers of truth, and both believe each other to be honest, then they should update on each other's opinions and quickly reach agreement. The very fact that a person believes something is Rational evidence that that something is true, and so this fact should be taken into account when forming your belief.
Outside of well-functioning prediction markets, Aumann agreement can probably only be approximated by careful deliberative discourse.  Thus, fostering effective deliberation should be seen as a key goal of Less Wrong.
A: Disagreement

Q: Doubt
A: The proper purpose of a doubt is to destroy its target belief if and only if it is false. Merely going around doubting things, without following up and investigating, doesn't help you get truer beliefs. The mere feeling of crushing uncertainty is not virtuous unto an aspiring rationalist; probability theory is the law that says we must be uncertain to the exact extent to which the evidence merits uncertainty.

Q: The proper purpose of a [...] is to destroy its target belief if and only if it is false. Merely going around [...]ing things, without following up and investigating, doesn't help you get truer beliefs. The mere feeling of crushing uncertainty is not virtuous unto an aspiring rationalist; probability theory is the law that says we must be uncertain to the exact extent to which the evidence merits uncertainty.
A: Doubt

Q: Egalitarianism
A: If you can't admit to yourself that you've done better than others - or if you're ashamed of wanting to do better than others - then the median will forever be your concrete wall, the place where you stop moving forward.  And what about people who are below average?  Do you dare say you intend to do better than them?  How prideful of you!
Maybe it's not healthy to pride yourself on doing better than someone else.  Personally I've found it to be a useful motivator, despite my principles, and I'll take all the useful motivation I can get.  Maybe that kind of competition is a zero-sum game, but then so is Go; it doesn't mean we should abolish that human activity, if people find it fun and it leads somewhere interesting.
But in any case, surely it isn't healthy to be ashamed of doing better.
—Tsuyoku vs. the Egalitarian Instinct
Egalitarianism is the idea that everyone should be considered equal. Equal in merit, equal in opportunity, equal in morality, and equal in achievement.
Dismissing egalitarianism is not opposed to humility, even though from the signaling perspective it seems to be opposed to modesty.

Q: If you can't admit to yourself that you've done better than others - or if you're ashamed of wanting to do better than others - then the median will forever be your concrete wall, the place where you stop moving forward.  And what about people who are below average?  Do you dare say you intend to do better than them?  How prideful of you!
Maybe it's not healthy to pride yourself on doing better than someone else.  Personally I've found it to be a useful motivator, despite my principles, and I'll take all the useful motivation I can get.  Maybe that kind of competition is a zero-sum game, but then so is Go; it doesn't mean we should abolish that human activity, if people find it fun and it leads somewhere interesting.
But in any case, surely it isn't healthy to be ashamed of doing better.
—Tsuyoku vs. the Egalitarian Instinct
[...] is the idea that everyone should be considered equal. Equal in merit, equal in opportunity, equal in morality, and equal in achievement.
Dismissing [...] is not opposed to humility, even though from the signaling perspective it seems to be opposed to modesty.
A: Egalitarianism

Q: Egan\'s law
A: It all adds up to normality.
—Greg Egan, Quarantine
Apples didn't stop falling when General Relativity supplanted Newtonian mechanics.  The purpose of a theory is to add up to observed reality, rather than something else.  Quantum mechanics is not some strange alternative universe, told as a story to delight us with its counterintuitiveness; quantum mechanics is where we have always lived.  Science sets out to answer the question "What adds up to normality?" and the answer turns out to be "Quantum mechanics adds up to normality."
A weaker extension of this principle applies to ethical and metaethical debates, which generally ought to end up explaining why you shouldn't eat babies, rather than why you should.

Q: It all adds up to normality.
—Greg Egan, Quarantine
Apples didn't stop falling when General Relativity supplanted Newtonian mechanics.  The purpose of a theory is to add up to observed reality, rather than something else.  Quantum mechanics is not some strange alternative universe, told as a story to delight us with its counterintuitiveness; quantum mechanics is where we have always lived.  Science sets out to answer the question "What adds up to normality?" and the answer turns out to be "Quantum mechanics adds up to normality."
A weaker extension of this principle applies to ethical and metaethical debates, which generally ought to end up explaining why you shouldn't eat babies, rather than why you should.
A: Egan\'s law

Q: Emotion
A: Relinquish the emotion which rests upon a mistaken belief, and seek to feel fully that emotion which fits the facts. If the iron approaches your face, and you believe it is hot, and it is cool, the Way opposes your fear. If the iron approaches your face, and you believe it is cool, and it is hot, the Way opposes your calm. Evaluate your beliefs first and then arrive at your emotions.
—Twelve Virtues of Rationality
Contrary to the stereotype, rationality doesn't mean denying emotion. When emotion is appropriate to the reality of the situation, it should be embraced; only when emotion isn't appropriate should it be suppressed.

Q: Relinquish the [...] which rests upon a mistaken belief, and seek to feel fully that [...] which fits the facts. If the iron approaches your face, and you believe it is hot, and it is cool, the Way opposes your fear. If the iron approaches your face, and you believe it is cool, and it is hot, the Way opposes your calm. Evaluate your beliefs first and then arrive at your [...]s.
—Twelve Virtues of Rationality
Contrary to the stereotype, rationality doesn't mean denying [...]. When [...] is appropriate to the reality of the situation, it should be embraced; only when [...] isn't appropriate should it be suppressed.
A: Emotion

Q: Epistemic hygiene
A: Epistemic hygiene consists of practices meant to allow accurate beliefs to spread within a community and keep less accurate or biased beliefs contained. The practices are meant to serve an analogous purpose to normal hygiene and sanitation in containing disease. "Good cognitive citizenship" is another phrase that has been proposed for this concept[1].

Q: [...] consists of practices meant to allow accurate beliefs to spread within a community and keep less accurate or biased beliefs contained. The practices are meant to serve an analogous purpose to normal hygiene and sanitation in containing disease. "Good cognitive citizenship" is another phrase that has been proposed for this concept[1].
A: Epistemic hygiene

Q: Epistemic luck
A: You would have different beliefs if certain events in your life were different. How should you react to this fact?

Q: You would have different beliefs if certain events in your life were different. How should you react to this fact?
A: Epistemic luck

Q: Error of crowds
A: The error of crowds is the idea that under some scoring rules, the average error becomes less than the error of the average, thus making the average belief tautologically worse than a belief of a random person. Compare this to the ideas of modesty argument and wisdom of the crowd.
A related idea is that a popular belief is likely to be wrong because the less popular ones couldn't maintain support if they were worse than the popular one.

Q: The [...] is the idea that under some scoring rules, the average error becomes less than the error of the average, thus making the average belief tautologically worse than a belief of a random person. Compare this to the ideas of modesty argument and wisdom of the crowd.
A related idea is that a popular belief is likely to be wrong because the less popular ones couldn't maintain support if they were worse than the popular one.
A: Error of crowds

Q: Ethical injunction
A: Ethical injunctions are rules not to do something even when you believe it's the right thing to do.  (That is, you refrain "even when your brain has computed it's the right thing to do", but this will just seem like "the right thing to do".)
For example, you shouldn't rob banks even if you plan to give the money to a good cause.
This is to protect you from your own cleverness (especially taking bad black swan bets), and the Corrupted hardware you're running on.
Related to the Metaethics sequence.
Sequence
Why Does Power Corrupt?
Power corrupts is well known folk wisdom. This post gives an evo-psych explanation. Corrupt behavior provides a fitness advantage, but signaling corruption makes it hard to get power. The cleanest way to not signal corruption is to honestly believe that one will not be corrupt. Thus the fittest strategy is to couple an honest desire to do good with a tendency to find the common abuses  of power pleasurable.
This post is not cross listed as a part of the listed main sequences.
Ends Don't Justify Means (Among Humans)
"The end does not justify the means" is just consequentialist reasoning at one meta-level up.  If a human starts thinking on the object level that the end justifies the means, this has awful consequences given our untrustworthy brains; therefore a human shouldn't think this way.  But it is all still ultimately consequentialism.  It's just reflective consequentialism, for beings who know that their moment-by-moment decisions are made by untrusted hardware.
This post is not cross listed as a part of the listed main sequences.
Entangled Truths, Contagious Lies
Most lies, in order to stand against rigorous investigation, would require additional lies about supporting facts. Since people do not know all aspects of all disciplines, the web of supporting lies will eventually entail making a claim that is self evidently false to someone with expert knowledge the liar does not possess. Only a god could lie to an AI.
Part of the Against Rationalization subsequence of How To Actually Change Your Mind
Protected From Myself
A more personal / reflective post in which Eliezer looks back and observes that his ethically motivated truthfulness has led to better outcomes than he would have achieved by lying. He proposes several reasons for this including that honesty makes it harder to sweep problems away forcing him to deal with them.
This post is not cross listed as a part of the listed main sequences.
Ethical Inhibitions
A speculative evo psych post reasoning that "ethical instincts" would have been adaptive in a context where people systemically underestimated the risks of getting caught ( see general overconfidence bias) and were punished heavily via exile from the tribe or outright death.
This post is not cross listed as a part of the listed main sequences.
Ethical Injunctions
Linking the previous posts in the sequence to the problem of AI, this post explores ethical injunctions as failsafe mechanisms in a self-modifying AI. A simple example is that if an AI in the takeoff phase decides at iteration N that it needs to deceive it programmers about its end goals, then the goals have likely drifted too far during the modification process. An injunction against deceiving the programmers will shut down the AI before it gets any worse. Further, the AI at step N-1 will hopefully have seen this itself and built the injunction into its next iteration. As humans with many subconscious biases, a choice to impose ethical injunctions on ourselves can serve as a similar failsafe.
This post is not cross listed as a part of the listed main sequences.
Prices or Bindings?
Certain opportunities to violate an injunction will only arise if the injunction exists; some planning a murder will only confess if he expects the priest not to testify. Thus the apparent gain from violating an injunction in a single case does not actually exist on a systemic level. If prospective murders know that priests makes exception for murders, then they won’t confess to the priest and the priest will not have the opportunity to make an exception. Injunctions that seem value destructive in single instance hypotheticals can be beneficial at a systemic level.
This post is not cross listed as a part of the listed main sequences.
Ethics Notes
This is a round-up of some of the more interesting and insightful comments to prior posts in the sequence with detailed responses brought to the front.
This post is not cross listed as a part of the listed main sequences.

Q: [...]s are rules not to do something even when you believe it's the right thing to do.  (That is, you refrain "even when your brain has computed it's the right thing to do", but this will just seem like "the right thing to do".)
For example, you shouldn't rob banks even if you plan to give the money to a good cause.
This is to protect you from your own cleverness (especially taking bad black swan bets), and the Corrupted hardware you're running on.
Related to the Metaethics sequence.
Sequence
Why Does Power Corrupt?
Power corrupts is well known folk wisdom. This post gives an evo-psych explanation. Corrupt behavior provides a fitness advantage, but signaling corruption makes it hard to get power. The cleanest way to not signal corruption is to honestly believe that one will not be corrupt. Thus the fittest strategy is to couple an honest desire to do good with a tendency to find the common abuses  of power pleasurable.
This post is not cross listed as a part of the listed main sequences.
Ends Don't Justify Means (Among Humans)
"The end does not justify the means" is just consequentialist reasoning at one meta-level up.  If a human starts thinking on the object level that the end justifies the means, this has awful consequences given our untrustworthy brains; therefore a human shouldn't think this way.  But it is all still ultimately consequentialism.  It's just reflective consequentialism, for beings who know that their moment-by-moment decisions are made by untrusted hardware.
This post is not cross listed as a part of the listed main sequences.
Entangled Truths, Contagious Lies
Most lies, in order to stand against rigorous investigation, would require additional lies about supporting facts. Since people do not know all aspects of all disciplines, the web of supporting lies will eventually entail making a claim that is self evidently false to someone with expert knowledge the liar does not possess. Only a god could lie to an AI.
Part of the Against Rationalization subsequence of How To Actually Change Your Mind
Protected From Myself
A more personal / reflective post in which Eliezer looks back and observes that his ethically motivated truthfulness has led to better outcomes than he would have achieved by lying. He proposes several reasons for this including that honesty makes it harder to sweep problems away forcing him to deal with them.
This post is not cross listed as a part of the listed main sequences.
Ethical Inhibitions
A speculative evo psych post reasoning that "ethical instincts" would have been adaptive in a context where people systemically underestimated the risks of getting caught ( see general overconfidence bias) and were punished heavily via exile from the tribe or outright death.
This post is not cross listed as a part of the listed main sequences.
Ethical Injunctions
Linking the previous posts in the sequence to the problem of AI, this post explores [...]s as failsafe mechanisms in a self-modifying AI. A simple example is that if an AI in the takeoff phase decides at iteration N that it needs to deceive it programmers about its end goals, then the goals have likely drifted too far during the modification process. An injunction against deceiving the programmers will shut down the AI before it gets any worse. Further, the AI at step N-1 will hopefully have seen this itself and built the injunction into its next iteration. As humans with many subconscious biases, a choice to impose [...]s on ourselves can serve as a similar failsafe.
This post is not cross listed as a part of the listed main sequences.
Prices or Bindings?
Certain opportunities to violate an injunction will only arise if the injunction exists; some planning a murder will only confess if he expects the priest not to testify. Thus the apparent gain from violating an injunction in a single case does not actually exist on a systemic level. If prospective murders know that priests makes exception for murders, then they won’t confess to the priest and the priest will not have the opportunity to make an exception. Injunctions that seem value destructive in single instance hypotheticals can be beneficial at a systemic level.
This post is not cross listed as a part of the listed main sequences.
Ethics Notes
This is a round-up of some of the more interesting and insightful comments to prior posts in the sequence with detailed responses brought to the front.
This post is not cross listed as a part of the listed main sequences.
A: Ethical injunction

Q: Everett branch
A: An Everett branch is one of the "worlds" in the many-worlds interpretation of quantum mechanics. On a naïve view, the world we perceive around us is all that we have reason to expect exists. However, clever experiments in quantum mechanics tell us that the universe is actually described by a mathematical object called a wavefunction, with our observable universe existing "side-by-side" with many alternatives or "parallel universes". It is difficult to explain the necessary concepts in just a few sentences, and true understanding requires mathematical grounding.

Q: An [...] is one of the "worlds" in the many-worlds interpretation of quantum mechanics. On a naïve view, the world we perceive around us is all that we have reason to expect exists. However, clever experiments in quantum mechanics tell us that the universe is actually described by a mathematical object called a wavefunction, with our observable universe existing "side-by-side" with many alternatives or "parallel universes". It is difficult to explain the necessary concepts in just a few sentences, and true understanding requires mathematical grounding.
A: Everett branch

Q: Evidence
A: Evidence for a given theory is the observation of an event that is more likely to occur if the theory is true than if it is false. (The event would be evidence against the theory if it is less likely if the theory is true.)
The likelihood ratio term  from Bayes' theorem is greater than 1 if the event B is evidence of the theory A, and less than 1 if the event is evidence against the theory.

Q: [...] for a given theory is the observation of an event that is more likely to occur if the theory is true than if it is false. (The event would be [...] against the theory if it is less likely if the theory is true.)
The likelihood ratio term  from Bayes' theorem is greater than 1 if the event B is [...] of the theory A, and less than 1 if the event is [...] against the theory.
A: Evidence

Q: Evidence of absence
A: Evidence of absence is evidence that allows you to conclude some phenomenon isn't there. It is often said that "absence of evidence is not evidence of absence". However, if evidence is expected, but not present, that is evidence of absence.

Q: [...] is evidence that allows you to conclude some phenomenon isn't there. It is often said that "absence of evidence is not [...]". However, if evidence is expected, but not present, that is [...].
A: Evidence of absence

Q: Evidential Decision Theory
A: Evidential Decision Theory is a branch of decision theory which advises an agent to act to place itself in a reference class with the most favorable expected outcome. It maximizes P(O|A), the expected utility of outcome O given that action A has been performed, which contrasts against Causal Decision Theory.
EDT is criticized for recommending courses of action with no causal relationship to desired outcomes. The difference between Evidential Decision Theory and Causal Decision Theory is highlighted by EDT giving the wrong answer to the Smoking lesion problem.
See also
Decision theory
Causal Decision Theory

Q: [...] is a branch of decision theory which advises an agent to act to place itself in a reference class with the most favorable expected outcome. It maximizes P(O|A), the expected utility of outcome O given that action A has been performed, which contrasts against Causal Decision Theory.
EDT is criticized for recommending courses of action with no causal relationship to desired outcomes. The difference between [...] and Causal Decision Theory is highlighted by EDT giving the wrong answer to the Smoking lesion problem.
See also
Decision theory
Causal Decision Theory
A: Evidential Decision Theory

Q: Evolution
A: The brainless, mindless optimization process responsible for the production of all biological life on Earth, including human beings.  Since the design signature of evolution is alien and counterintuitive, it takes some study to get to know your accidental Creator.
Contents
1 See also
2 Blog posts (sequence)
3 Blog posts (other)
4 External links
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }

Q: The brainless, mindless optimization process responsible for the production of all biological life on Earth, including human beings.  Since the design signature of [...] is alien and counterintuitive, it takes some study to get to know your accidental Creator.
Contents
1 See also
2 Blog posts (sequence)
3 Blog posts (other)
4 External links
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
A: Evolution

Q: Evolutionary psychology
A: Evolution, the origin of all biological life on Earth, does not work like humans do, and does not design things the way a human engineer would.  This blind idiot god is also the source and patterner of human beings.  "Nothing in biology makes sense except in the light of evolution," said Theodosius Dobzhansky.  Humans brains are also biology, and nothing about our thinking makes sense except in the light of evolution.
Consider, for example, the following tale:
A man and a woman meet in a bar.  The man is attracted to her clear complexion, which would have been fertility cues in the ancestral environment, but which in this case result from makeup and a bra.  This does not bother the man; he just likes the way she looks.  His clear-complexion-detecting neural circuitry does not know that its purpose is to detect fertility, any more than the atoms in his hand contain tiny little XML tags reading "<purpose>pick things up</purpose>".  The woman is attracted to his confident smile and firm manner, cues to high status, which in the ancestral environment would have signified the ability to provide resources for children.  She plans to use birth control, but her confident-smile-detectors don't know this any more than a toaster knows its designer intended it to make toast.  She's not concerned philosophically with the meaning of this rebellion, because her brain is a creationist and denies vehemently that evolution exists.  He's not concerned philosophically with the meaning of this rebellion, because he just wants to get laid.  They go to a hotel, and undress.  He puts on a condom, because he doesn't want kids, just the dopamine-noradrenaline rush of sex, which reliably produced offspring 50,000 years ago when it was an invariant feature of the ancestral environment that condoms did not exist.  They have sex, and shower, and go their separate ways.  The main objective consequence is to keep the bar and the hotel and condom-manufacturer in business; which was not the cognitive purpose in their minds, and has virtually nothing to do with the key statistical regularities of reproduction 50,000 years ago which explain how they got the genes that built their brains that executed all this behavior.
This only makes sense in the light of evolution as a designer - that we are poorly optimized to reproduce by a blind and unforesightful god.
The idea of evolution as the idiot designer of humans - that our brains are not consistently well-designed - is a key element of many of the explanations of human errors that appear on this website.
Some of the key ideas of evolutionary psychology are these:

Q: Evolution, the origin of all biological life on Earth, does not work like humans do, and does not design things the way a human engineer would.  This blind idiot god is also the source and patterner of human beings.  "Nothing in biology makes sense except in the light of evolution," said Theodosius Dobzhansky.  Humans brains are also biology, and nothing about our thinking makes sense except in the light of evolution.
Consider, for example, the following tale:
A man and a woman meet in a bar.  The man is attracted to her clear complexion, which would have been fertility cues in the ancestral environment, but which in this case result from makeup and a bra.  This does not bother the man; he just likes the way she looks.  His clear-complexion-detecting neural circuitry does not know that its purpose is to detect fertility, any more than the atoms in his hand contain tiny little XML tags reading "<purpose>pick things up</purpose>".  The woman is attracted to his confident smile and firm manner, cues to high status, which in the ancestral environment would have signified the ability to provide resources for children.  She plans to use birth control, but her confident-smile-detectors don't know this any more than a toaster knows its designer intended it to make toast.  She's not concerned philosophically with the meaning of this rebellion, because her brain is a creationist and denies vehemently that evolution exists.  He's not concerned philosophically with the meaning of this rebellion, because he just wants to get laid.  They go to a hotel, and undress.  He puts on a condom, because he doesn't want kids, just the dopamine-noradrenaline rush of sex, which reliably produced offspring 50,000 years ago when it was an invariant feature of the ancestral environment that condoms did not exist.  They have sex, and shower, and go their separate ways.  The main objective consequence is to keep the bar and the hotel and condom-manufacturer in business; which was not the cognitive purpose in their minds, and has virtually nothing to do with the key statistical regularities of reproduction 50,000 years ago which explain how they got the genes that built their brains that executed all this behavior.
This only makes sense in the light of evolution as a designer - that we are poorly optimized to reproduce by a blind and unforesightful god.
The idea of evolution as the idiot designer of humans - that our brains are not consistently well-designed - is a key element of many of the explanations of human errors that appear on this website.
Some of the key ideas of [...] are these:
A: Evolutionary psychology

Q: Existential risk
A: An existential risk is a risk posing permanent consequences to humanity which can never be undone. In Nick Bostrom's 2002 article, he defined an existential risk as:
One where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.
Bostrom (2002) proposes a series of classifications for existential risks:
Bangs - Earthly intelligent life is extinguished relatively suddenly by any cause; the prototypical end of humanity. Examples of bangs include deliberate or accidental misuse of nanotechnology, nuclear holocaust, the end of our simulation, or an unfriendly AI.
Crunches - The potential humanity had to enhance itself indefinitely is forever eliminated, although humanity continues. Possible crunches include an exhaustion of resources, social or governmental pressure ending technological development, and even future technological development proving an unsurpassable challenge before the creation of a superintelligence.
Shrieks - Humanity enhances itself, but explores only a narrow portion of its desirable possibilities. As the criteria for desirability haven't been defined yet, this category is mainly undefined. However, a flawed friendly AI incorrectly interpreting our values, a superhuman upload deciding its own values and imposing them on the rest of humanity, and an intolerant government outlawing social progress would certainly qualify.
Whimpers - Though humanity is enduring, only a fraction of our potential is ever achieved. Spread across the galaxy and expanding at near light-speed, we might find ourselves doomed by ours or another being's catastrophic physics experimentation, destroying reality at light-speed. A prolonged galactic war leading to our extinction or severe limitation would also be a whimper. More darkly, humanity might develop until its values were disjoint with ours today, making their civilization worthless by present values.
Existential risks present a unique challenge because of their irreversible nature. We will never, by definition, experience and survive an existential risk and so cannot learn from our mistakes. As we have no past experience with existential risks, they are black swans.  Since existential disasters cannot be recovered from, their cost is not just the the dead, but every descendent who would have been born.

Q: An [...] is a risk posing permanent consequences to humanity which can never be undone. In Nick Bostrom's 2002 article, he defined an [...] as:
One where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.
Bostrom (2002) proposes a series of classifications for [...]s:
Bangs - Earthly intelligent life is extinguished relatively suddenly by any cause; the prototypical end of humanity. Examples of bangs include deliberate or accidental misuse of nanotechnology, nuclear holocaust, the end of our simulation, or an unfriendly AI.
Crunches - The potential humanity had to enhance itself indefinitely is forever eliminated, although humanity continues. Possible crunches include an exhaustion of resources, social or governmental pressure ending technological development, and even future technological development proving an unsurpassable challenge before the creation of a superintelligence.
Shrieks - Humanity enhances itself, but explores only a narrow portion of its desirable possibilities. As the criteria for desirability haven't been defined yet, this category is mainly undefined. However, a flawed friendly AI incorrectly interpreting our values, a superhuman upload deciding its own values and imposing them on the rest of humanity, and an intolerant government outlawing social progress would certainly qualify.
Whimpers - Though humanity is enduring, only a fraction of our potential is ever achieved. Spread across the galaxy and expanding at near light-speed, we might find ourselves doomed by ours or another being's catastrophic physics experimentation, destroying reality at light-speed. A prolonged galactic war leading to our extinction or severe limitation would also be a whimper. More darkly, humanity might develop until its values were disjoint with ours today, making their civilization worthless by present values.
[...]s present a unique challenge because of their irreversible nature. We will never, by definition, experience and survive an [...] and so cannot learn from our mistakes. As we have no past experience with [...]s, they are black swans.  Since existential disasters cannot be recovered from, their cost is not just the the dead, but every descendent who would have been born.
A: Existential risk

Q: Expected utility
A: Expected utility is the expected value in terms of the utility produced by an action. Each possible consequence of the action is assigned a utility by a utility function and weighed by the probability of that outcome occurring. The sum of these weighed utilities is defined to be the action's expected utility.
Von Neumann and Morgenstern proved the expected utility theorem, which says that when a rational agent chooses between different "gambles" (probability distributions over outcomes), the utility of such a gamble can always be seen as the expected utility of the gamble's outcome.
Humans, of course, are a different story.

Q: [...] is the expected value in terms of the utility produced by an action. Each possible consequence of the action is assigned a utility by a utility function and weighed by the probability of that outcome occurring. The sum of these weighed utilities is defined to be the action's [...].
Von Neumann and Morgenstern proved the [...] theorem, which says that when a rational agent chooses between different "gambles" (probability distributions over outcomes), the utility of such a gamble can always be seen as the [...] of the gamble's outcome.
Humans, of course, are a different story.
A: Expected utility

Q: Expected value
A: The expected value or expectation is the (weighted) average of all the possible outcomes of an event, weighed by their probability. For example, when you roll a die, the expected value is (1+2+3+4+5+6)/6 = 3.5.
(Since a die doesn't even have a face that says 3.5, this illustrates that very often, the "expected value" isn't a value you actually expect.)

Q: The [...] or expectation is the (weighted) average of all the possible outcomes of an event, weighed by their probability. For example, when you roll a die, the [...] is (1+2+3+4+5+6)/6 = 3.5.
(Since a die doesn't even have a face that says 3.5, this illustrates that very often, the "[...]" isn't a value you actually expect.)
A: Expected value

Q: Exploratory engineering
A: Exploratory engineering is a process for understanding a subset of the potential capabilities of future levels of technology. This process resembles the first phase of standard design engineering (termed conceptual engineering, or conceptual design), but it serves a different purpose:
In standard engineering, design leads to the manufacturing of a product.
In exploratory engineering, design leads to understanding of what a future manufacturing process could produce.
Exploratory engineering allows to show how some capabilities that intuitively seem absurd may actually be achieved in the future.

Q: [...] is a process for understanding a subset of the potential capabilities of future levels of technology. This process resembles the first phase of standard design engineering (termed conceptual engineering, or conceptual design), but it serves a different purpose:
In standard engineering, design leads to the manufacturing of a product.
In [...], design leads to understanding of what a future manufacturing process could produce.
[...] allows to show how some capabilities that intuitively seem absurd may actually be achieved in the future.
A: Exploratory engineering

Q: Extraordinary evidence
A: Extraordinary evidence is evidence that turns an a priori highly unlikely event into an a posteriori likely event.

Q: [...] is evidence that turns an a priori highly unlikely event into an a posteriori likely event.
A: Extraordinary evidence

Q: Friendly artificial intelligence
A: Friendly Artificial Intelligence (FAI) has two meanings. Its general meaning refers to any artificial general intelligence that has a positive rather than negative effect on humanity. The more specific meaning refers to the kinds of AGI designs which Eliezer Yudkowsky argues to be the only ones that can be expected to have a positive effect. The rest of this article uses the term in its more general sense.
In this context, Friendly AI also refers to the field of knowledge required to build such an AI. Note that Friendly (capital-F) is being used as a term of art, referring specifically to AIs that protect humans and humane values; an FAI need not be "friendly" in the conventional sense and need not even be sentient. Any AGI that is not friendly is said to be Unfriendly.
AI that underwent an intelligence explosion could exert unprecedented optimization power over its future; therefore, a Friendly AI could very well create an unimaginably good future. Conversely, an Unfriendly AI could represent an existential risk: destroying all humans, not out of hostility, but as a side effect of trying to do something entirely different. Just because an AI has the means to do something, doesn't mean it will.
Requiring Friendliness doesn't make AGI any easier, and almost certainly makes it harder. Most approaches to AGI aren't amenable to implementing precise goals, and so don't even constitute subprojects for FAI, leading to Unfriendly AI as the only possible "successful" outcome. Specifying Friendliness also presents unique technical challenges: humane moral value is very complex; a lot of seemingly simple-sounding moral concepts conceal hidden complexity not "inherent" in the universe itself. It is likely impossible to specify humane values by explicitly programming them in, one needs a technique for extracting them automatically.

Q: Friendly Artificial Intelligence (FAI) has two meanings. Its general meaning refers to any artificial general intelligence that has a positive rather than negative effect on humanity. The more specific meaning refers to the kinds of AGI designs which Eliezer Yudkowsky argues to be the only ones that can be expected to have a positive effect. The rest of this article uses the term in its more general sense.
In this context, Friendly AI also refers to the field of knowledge required to build such an AI. Note that Friendly (capital-F) is being used as a term of art, referring specifically to AIs that protect humans and humane values; an FAI need not be "friendly" in the conventional sense and need not even be sentient. Any AGI that is not friendly is said to be Unfriendly.
AI that underwent an intelligence explosion could exert unprecedented optimization power over its future; therefore, a Friendly AI could very well create an unimaginably good future. Conversely, an Unfriendly AI could represent an existential risk: destroying all humans, not out of hostility, but as a side effect of trying to do something entirely different. Just because an AI has the means to do something, doesn't mean it will.
Requiring Friendliness doesn't make AGI any easier, and almost certainly makes it harder. Most approaches to AGI aren't amenable to implementing precise goals, and so don't even constitute subprojects for FAI, leading to Unfriendly AI as the only possible "successful" outcome. Specifying Friendliness also presents unique technical challenges: humane moral value is very complex; a lot of seemingly simple-sounding moral concepts conceal hidden complexity not "inherent" in the universe itself. It is likely impossible to specify humane values by explicitly programming them in, one needs a technique for extracting them automatically.
A: Friendly artificial intelligence

Q: Fake simplicity
A: Explanations exist; they have existed for all time; there is always a well-known solution to every human problem — neat, plausible, and wrong.

Q: Explanations exist; they have existed for all time; there is always a well-known solution to every human problem — neat, plausible, and wrong.
A: Fake simplicity

Q: Fallacy
A: A Fallacy is the failure to apply logic to a line of reasoning that renders an argument invalid. The ability to recognize a variety of different forms of fallacy can be useful when analyzing arguments made by others and help avoid making logical errors when forming one’s own beliefs.
The structure of a logical argument should follow a certain basic structure. They begin with one or more premises, which act as the arguments starting point and then by applying principles of logic come to a valid conclusion.
If A = B and B = C then we can conclude A=C.
It is the failure to apply logic rigorously that leads to fallacious arguments.
Fallacies are divided into formal and informal groups. A formal fallacy has a flaw in the logical structure of the argument which renders the argument invalid, whilst an informal fallacy has a logical form, but is false due to the characteristics of its premises, or its justification structure.
Contents
1 Examples
2 False Accusations of Fallaciousness
3 External Links
4 See Also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Examples
There are dozens of different forms of fallacy some of the most common include:-
Ad hominem - where the character of the individual making the opposing argument is attacked, rather than the argument itself. e.g “He’s old, fat and bald. There’s no way he would make a good President.”
False dilemma  - where the argument proposes only a limited number of possible choices, when a variety of other options may be available. e.g “All people who commit murder should be executed, otherwise once they are released from prison they will murder again.”
Post hoc ergo propter hoc - asserting that because event B followed event A, event A caused event B. “My team won after I wore my new shirt, therefore my new shirt is a lucky shirt.”
Straw man - where an argument takes the form of a deliberately weak or inaccurate description  of the counter-argument rendering it easy to “burn down”. e.g “All modern artists do is slap a bit of paint randomly on the canvas without any skill or understanding, if that is art then my three-year old must be genius.”
An appeal to Authority - where an attempt to bolster an argument is made by using well known figure. e.g “We should not go to war with that country because Bono doesn’t want us to.”
False Accusations of Fallaciousness
Matters are further complicated by arguing parties incorrectly claiming that an assertion is false due to a fallacy. For example, if one party was to declare  “Albert Einstein has claimed that time and space are relative qualities of the Universe.”, another party might responded by saying that this is an ‘’’argument from authority’’’. However, Albert Einstein’s claims are based on highly detailed mathematical models that identify him as an expert in this field of inquiry, rather than a casual observer.
Equally, if somebody was to make the ad hominem attack “That guy can’t play for the Lakers, he’s only 3 feet tall!”, it would be a valid point as the characteristics of the individual do have a bearing on his ability to play basketball.
Whilst it is possible to overcome these issues by clarifying one's point e.g "Einstein's equations demonstrate that time and space are relative qualities of the Universe" or "He's only 3 feet tall which will make it much harder for him to score" - the necessity to validate each and every aspect of every premise would ultimately mean every argument would have to infinitely regress into explaining all aspects of the entire Universe. So certain commonsense, shared assumptions do need to be made in order to communicate an idea.
However it remains important to consider the possibility that under certain circumstances the structure of an argument may take the form of a recognized fallacy, but under closer examination may be perfectly valid.

Q: A [...] is the failure to apply logic to a line of reasoning that renders an argument invalid. The ability to recognize a variety of different forms of [...] can be useful when analyzing arguments made by others and help avoid making logical errors when forming one’s own beliefs.
The structure of a logical argument should follow a certain basic structure. They begin with one or more premises, which act as the arguments starting point and then by applying principles of logic come to a valid conclusion.
If A = B and B = C then we can conclude A=C.
It is the failure to apply logic rigorously that leads to fallacious arguments.
Fallacies are divided into formal and informal groups. A formal [...] has a flaw in the logical structure of the argument which renders the argument invalid, whilst an informal [...] has a logical form, but is false due to the characteristics of its premises, or its justification structure.
Contents
1 Examples
2 False Accusations of Fallaciousness
3 External Links
4 See Also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Examples
There are dozens of different forms of [...] some of the most common include:-
Ad hominem - where the character of the individual making the opposing argument is attacked, rather than the argument itself. e.g “He’s old, fat and bald. There’s no way he would make a good President.”
False dilemma  - where the argument proposes only a limited number of possible choices, when a variety of other options may be available. e.g “All people who commit murder should be executed, otherwise once they are released from prison they will murder again.”
Post hoc ergo propter hoc - asserting that because event B followed event A, event A caused event B. “My team won after I wore my new shirt, therefore my new shirt is a lucky shirt.”
Straw man - where an argument takes the form of a deliberately weak or inaccurate description  of the counter-argument rendering it easy to “burn down”. e.g “All modern artists do is slap a bit of paint randomly on the canvas without any skill or understanding, if that is art then my three-year old must be genius.”
An appeal to Authority - where an attempt to bolster an argument is made by using well known figure. e.g “We should not go to war with that country because Bono doesn’t want us to.”
False Accusations of Fallaciousness
Matters are further complicated by arguing parties incorrectly claiming that an assertion is false due to a [...]. For example, if one party was to declare  “Albert Einstein has claimed that time and space are relative qualities of the Universe.”, another party might responded by saying that this is an ‘’’argument from authority’’’. However, Albert Einstein’s claims are based on highly detailed mathematical models that identify him as an expert in this field of inquiry, rather than a casual observer.
Equally, if somebody was to make the ad hominem attack “That guy can’t play for the Lakers, he’s only 3 feet tall!”, it would be a valid point as the characteristics of the individual do have a bearing on his ability to play basketball.
Whilst it is possible to overcome these issues by clarifying one's point e.g "Einstein's equations demonstrate that time and space are relative qualities of the Universe" or "He's only 3 feet tall which will make it much harder for him to score" - the necessity to validate each and every aspect of every premise would ultimately mean every argument would have to infinitely regress into explaining all aspects of the entire Universe. So certain commonsense, shared assumptions do need to be made in order to communicate an idea.
However it remains important to consider the possibility that under certain circumstances the structure of an argument may take the form of a recognized [...], but under closer examination may be perfectly valid.
A: Fallacy

Q: Fallacy of gray
A: The fallacy of gray is a belief that because nothing is certain, everything is equally uncertain. One who commits this fallacy may reply to the statement that probability of winning a lottery is only one in a million by saying: "There's still a chance, right?"
This fallacy is invoked by those who wish to attack a well performing system (e.g. Science), by saying that it is still imperfect. This then excuses the imperfections of the invoker's chosen system. For example
"Science is based on faith too!"

Q: The [...] is a belief that because nothing is certain, everything is equally uncertain. One who commits this fallacy may reply to the statement that probability of winning a lottery is only one in a million by saying: "There's still a chance, right?"
This fallacy is invoked by those who wish to attack a well performing system (e.g. Science), by saying that it is still imperfect. This then excuses the imperfections of the invoker's chosen system. For example
"Science is based on faith too!"
A: Fallacy of gray

Q: False dilemma
A: A false dilemma (false dichotomy) occurs when only two options are considered, when there may in fact be many.

Q: A [...] (false dichotomy) occurs when only two options are considered, when there may in fact be many.
A: False dilemma

Q: Filtered evidence
A: Filtered evidence is evidence that was selected for the purpose of proving (disproving) a hypothesis. Filtered evidence may be highly misleading, but still it can be useful, if considered with care.

Q: [...] is evidence that was selected for the purpose of proving (disproving) a hypothesis. [...] may be highly misleading, but still it can be useful, if considered with care.
A: Filtered evidence

Q: Intelligence explosion
A: Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultra-intelligent machine could design even better machines; there would then unquestionably be an "intelligence explosion," and the intelligence of man would be left far behind.
—I.J. Good, "Speculations Concerning the First Ultraintelligent Machine"
Intelligence explosion is the idea of a positive feedback loop in which an intelligence is making itself smarter, thus getting better at making itself even smarter. A strong version of this idea suggests that once the positive feedback starts to play a role, it will lead to a dramatic leap in capability very quickly. Depending on the mechanism underlying the feedback loop, the transition may take years or hours (intelligence explosion doesn't necessarily mean recursively self-improving AIs, other options include brain-computer interfaces or even genetic engineering). At some point technological progress drops into the characteristic timescale of transistors (or super-transistors) rather than human neurons, and the ascent rapidly surges upward and creates superintelligence (minds orders of magnitude more powerful than human) before it hits physical limits.
Contents
1 Published arguments
2 Hypothetical path
3 Blog posts
4 See also
5 External links
6 References
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Published arguments
Philosopher David Chalmers published a significant analysis of the Singularity, focusing on intelligence explosions, in Journal of Consciousness Studies. His analysis of how they could occur defends the likelihood of an intelligence explosion. He also discusses the nature of general intelligence, and possible obstacles to a singularity. A good deal of discussion is given to the dangers of an intelligence explosion, and Chalmers concludes that we must negotiate it very carefully by building the correct values into the initial AIs.
Luke Muehlhauser (lukeprog) and Anna Salamon (AnnaSalamon) argue in Intelligence Explosion: Evidence and Import in detail that an intelligence explosion is very likely within 100 years, and extremely critical in determining the future. They trace the implications of many types of upcoming technologies, and point out the feedback loops present in them. This leads them to deduce that an above-human level AI will almost certainly lead to an intelligence explosion. They conclude with recommendations for bringing about a safe intelligence explosion.
Hypothetical path
The following is a common example of a possible path for an AI to bring about an intelligence explosion. First, the AI is smart enough to conclude that inventing molecular nanotechnology will be of greatest benefit to it. Its first act of recursive self-improvement is to gain access to other computers over the internet. This extra computational ability increases the depth and breadth of its search processes. It then uses gained knowledge of material physics and a distributed computing program to invent the first general assembler nanomachine. Then it uses some manufacturing technology, accessible from the internet, to build and deploy the nanotech. It programs the nanotech to turn a large section of bedrock into a supercomputer. This is its second act of recursive self-improvement, only possible because of the first. Then it could use this enormous computing power to consider hundreds of alternative decision algorithms, better computing structures, et cetera.

Q: Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultra-intelligent machine could design even better machines; there would then unquestionably be an "[...]," and the intelligence of man would be left far behind.
—I.J. Good, "Speculations Concerning the First Ultraintelligent Machine"
[...] is the idea of a positive feedback loop in which an intelligence is making itself smarter, thus getting better at making itself even smarter. A strong version of this idea suggests that once the positive feedback starts to play a role, it will lead to a dramatic leap in capability very quickly. Depending on the mechanism underlying the feedback loop, the transition may take years or hours ([...] doesn't necessarily mean recursively self-improving AIs, other options include brain-computer interfaces or even genetic engineering). At some point technological progress drops into the characteristic timescale of transistors (or super-transistors) rather than human neurons, and the ascent rapidly surges upward and creates superintelligence (minds orders of magnitude more powerful than human) before it hits physical limits.
Contents
1 Published arguments
2 Hypothetical path
3 Blog posts
4 See also
5 External links
6 References
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Published arguments
Philosopher David Chalmers published a significant analysis of the Singularity, focusing on [...]s, in Journal of Consciousness Studies. His analysis of how they could occur defends the likelihood of an [...]. He also discusses the nature of general intelligence, and possible obstacles to a singularity. A good deal of discussion is given to the dangers of an [...], and Chalmers concludes that we must negotiate it very carefully by building the correct values into the initial AIs.
Luke Muehlhauser (lukeprog) and Anna Salamon (AnnaSalamon) argue in Intelligence Explosion: Evidence and Import in detail that an [...] is very likely within 100 years, and extremely critical in determining the future. They trace the implications of many types of upcoming technologies, and point out the feedback loops present in them. This leads them to deduce that an above-human level AI will almost certainly lead to an [...]. They conclude with recommendations for bringing about a safe [...].
Hypothetical path
The following is a common example of a possible path for an AI to bring about an [...]. First, the AI is smart enough to conclude that inventing molecular nanotechnology will be of greatest benefit to it. Its first act of recursive self-improvement is to gain access to other computers over the internet. This extra computational ability increases the depth and breadth of its search processes. It then uses gained knowledge of material physics and a distributed computing program to invent the first general assembler nanomachine. Then it uses some manufacturing technology, accessible from the internet, to build and deploy the nanotech. It programs the nanotech to turn a large section of bedrock into a supercomputer. This is its second act of recursive self-improvement, only possible because of the first. Then it could use this enormous computing power to consider hundreds of alternative decision algorithms, better computing structures, et cetera.
A: Intelligence explosion

Q: Free-floating belief
A: A free-floating belief is a belief that both doesn't follow from observations and doesn't restrict which experiences to anticipate. It is both unfounded and useless.

Q: A [...] is a belief that both doesn't follow from observations and doesn't restrict which experiences to anticipate. It is both unfounded and useless.
A: Free-floating belief

Q: Free will
A: One of the easiest hard questions, as millennia-old philosophical dilemmas go. Though this impossible question is fully and completely dissolved on Less Wrong, aspiring reductionists should try to solve it on their own.
Non-spoiler posts
The following posts can be read to set up the problem of "free will" and what constitutes a good solution from a reductionist perspective, without entirely giving away the solution. (When Yudkowsky wrote these posts, he thought he could get away with just leaving it as a practice problem, and some of the posts state that the problem will be left open. However Yudkowsky did eventually find that he needed to write out the whole solution.)
How An Algorithm Feels From Inside (see also the wiki page)
Dissolving the Question - this is where the "free will" puzzle is explicitly posed, along with criteria for what does and does not constitute a satisfying answer.
Wrong Questions
Righting a Wrong Question
For spoiler posts see free will (solution).

Q: One of the easiest hard questions, as millennia-old philosophical dilemmas go. Though this impossible question is fully and completely dissolved on Less Wrong, aspiring reductionists should try to solve it on their own.
Non-spoiler posts
The following posts can be read to set up the problem of "[...]" and what constitutes a good solution from a reductionist perspective, without entirely giving away the solution. (When Yudkowsky wrote these posts, he thought he could get away with just leaving it as a practice problem, and some of the posts state that the problem will be left open. However Yudkowsky did eventually find that he needed to write out the whole solution.)
How An Algorithm Feels From Inside (see also the wiki page)
Dissolving the Question - this is where the "[...]" puzzle is explicitly posed, along with criteria for what does and does not constitute a satisfying answer.
Wrong Questions
Righting a Wrong Question
For spoiler posts see [...] (solution).
A: Free will

Q: Free will (solution)
A: Since free will is about as easy as a philosophical problem in reductionism can get, while still appearing "impossible" to at least some philosophers, it makes a good exercise for aspiring reductionists, which they should try on their own - see the main page on free will.
These posts should not be read until having made a very serious effort on your own.
Blog posts
(Items in italics can be skipped if in a rush.)

Q: Since free will is about as easy as a philosophical problem in reductionism can get, while still appearing "impossible" to at least some philosophers, it makes a good exercise for aspiring reductionists, which they should try on their own - see the main page on free will.
These posts should not be read until having made a very serious effort on your own.
Blog posts
(Items in italics can be skipped if in a rush.)
A: Free will (solution)

Q: Friedman unit
A: The Wikipedia entry on Friedman Units tracks over 30 different cases between 2003 and 2007 in which someone labeled the "next six months" as the "critical period in Iraq".  Apparently one of the worst offenders is journalist Thomas Friedman after whom the unit was named (8 different predictions in 4 years).  In similar news, some of my colleagues in Artificial Intelligence (you know who you are) have been predicting the spectacular success of their projects in "3-5 years" for as long as I've known them, that is, since at least 2000.

Q: The Wikipedia entry on Friedman Units tracks over 30 different cases between 2003 and 2007 in which someone labeled the "next six months" as the "critical period in Iraq".  Apparently one of the worst offenders is journalist Thomas Friedman after whom the unit was named (8 different predictions in 4 years).  In similar news, some of my colleagues in Artificial Intelligence (you know who you are) have been predicting the spectacular success of their projects in "3-5 years" for as long as I've known them, that is, since at least 2000.
A: Friedman unit

Q: Fully general counterargument
A: A fully general counterargument is an argument which can be used to discount any conclusion the arguer does not like. Being in possession of such an argument leads to irrationality because it allows the arguer to avoid updating their beliefs in the light of new evidence.
Knowledge of cognitive biases can itself allow someone to form fully general counterarguments ("you're just saying that because you're exhibiting X bias").

Q: A [...] is an argument which can be used to discount any conclusion the arguer does not like. Being in possession of such an argument leads to irrationality because it allows the arguer to avoid updating their beliefs in the light of new evidence.
Knowledge of cognitive biases can itself allow someone to form [...]s ("you're just saying that because you're exhibiting X bias").
A: Fully general counterargument

Q: Fun theory
A: Fun theory addresses the following problem: If we can arrange to live for a very long time with greatly increased  intelligence and physical abilities, how can we continue to have any fun?
When we have intellects that can deeply comprehend every movie, novel, and concert ever created in an instant, and instantly solve any problem which might challenge a human today; or when superhumanly intelligent machines take care of our every need--then what fun will remain?
Contents
1 The risk of boredom
2 Fun Theory and complex values
3 Boredom
4 Relation to Friendly AI
5 External links
6 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
The risk of boredom
Unless we can answer this question, we might be faced by endless boredom. Alternatively, we could eliminate boredom from our mental makeup. We could change our mental architecture so that even the most trivial entertainments keep us amused forever. This too, is a future that we don't want.
These scenarios could be seen as an argument against key hopes of transhumanism for the improvement of the human condition, including lifespan extension, human intelligence enhancement,  physical enhancement, and Friendly AI.
Transhumanists work towards a much better human future--a Utopia--but, as George Orwell  aptly described it  Utopians of all stripes, Socialist, Enlightenment, or Christian, have generally been unable to imagine futures where anyone would actually want to live.
It is a commonplace that the Christian Heaven, as usually portrayed, would attract nobody. Almost all Christian writers dealing with Heaven either say frankly that it is indescribable or conjure up a vague picture of gold, precious stones, and the endless singing of hymns... [W]hat it could not do was to describe a condition in which the ordinary human being actively wanted to be.
Fun Theory and complex values
A key insight of Fun Theory, in its current embryonic form, is that eudaimonia is complicated--there are many properties which contribute to a life worth living. To experience a fulfilled life, we humans require many things: Aesthetic stimulation, pleasure, love, social interaction, learning, challenge, and much more.
It is a common mistake in discussion of future AI  extract one element of the human preferences and advocate that it  alone be maximized. This would lead to neglect of all the other values. For example, we simply optimize for pleasure or happiness, we will "wirehead"--stimulate the relevant parts of our brain and experience bliss for eternity, but pursue no other experiences. If almost any element of our value system  is absent the human future will likely be very unpleasant.
Enhanced humans will also have the value system of humans today, but we may choose to change it as we self-enhance. We may want to alter our own value system, by eliminating values, like bloodlust, which on reflection we wish were absent. But there are many values which we, on reflection want to keep, and since we humans have no basis for a value system other than our current value system, Fun Theory must seek to maximize the value system that we have, rather than inventing new values.
Boredom
Among the improvements  to the human condition that we want to work for, intelligence enhancement is among the most important. can work towards. Because of this, among other values that must be satisfied, Fun Theory puts a particular emphasis on the values opposed to boredom: curiosity, learning, and intellectual exploration. The open question is whether the universe will offer, or we ourselves can create, ever more complex and sophisticated opportunities to delight and challenge ever more powerful minds.
Relation to Friendly AI
An artificial general intelligence which is created to help humanity (Friendly AI), and which grows to be much more powerful than us, must have as its goal the promotion of the human value system.

Q: [...] addresses the following problem: If we can arrange to live for a very long time with greatly increased  intelligence and physical abilities, how can we continue to have any fun?
When we have intellects that can deeply comprehend every movie, novel, and concert ever created in an instant, and instantly solve any problem which might challenge a human today; or when superhumanly intelligent machines take care of our every need--then what fun will remain?
Contents
1 The risk of boredom
2 Fun Theory and complex values
3 Boredom
4 Relation to Friendly AI
5 External links
6 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
The risk of boredom
Unless we can answer this question, we might be faced by endless boredom. Alternatively, we could eliminate boredom from our mental makeup. We could change our mental architecture so that even the most trivial entertainments keep us amused forever. This too, is a future that we don't want.
These scenarios could be seen as an argument against key hopes of transhumanism for the improvement of the human condition, including lifespan extension, human intelligence enhancement,  physical enhancement, and Friendly AI.
Transhumanists work towards a much better human future--a Utopia--but, as George Orwell  aptly described it  Utopians of all stripes, Socialist, Enlightenment, or Christian, have generally been unable to imagine futures where anyone would actually want to live.
It is a commonplace that the Christian Heaven, as usually portrayed, would attract nobody. Almost all Christian writers dealing with Heaven either say frankly that it is indescribable or conjure up a vague picture of gold, precious stones, and the endless singing of hymns... [W]hat it could not do was to describe a condition in which the ordinary human being actively wanted to be.
Fun Theory and complex values
A key insight of Fun Theory, in its current embryonic form, is that eudaimonia is complicated--there are many properties which contribute to a life worth living. To experience a fulfilled life, we humans require many things: Aesthetic stimulation, pleasure, love, social interaction, learning, challenge, and much more.
It is a common mistake in discussion of future AI  extract one element of the human preferences and advocate that it  alone be maximized. This would lead to neglect of all the other values. For example, we simply optimize for pleasure or happiness, we will "wirehead"--stimulate the relevant parts of our brain and experience bliss for eternity, but pursue no other experiences. If almost any element of our value system  is absent the human future will likely be very unpleasant.
Enhanced humans will also have the value system of humans today, but we may choose to change it as we self-enhance. We may want to alter our own value system, by eliminating values, like bloodlust, which on reflection we wish were absent. But there are many values which we, on reflection want to keep, and since we humans have no basis for a value system other than our current value system, Fun Theory must seek to maximize the value system that we have, rather than inventing new values.
Boredom
Among the improvements  to the human condition that we want to work for, intelligence enhancement is among the most important. can work towards. Because of this, among other values that must be satisfied, Fun Theory puts a particular emphasis on the values opposed to boredom: curiosity, learning, and intellectual exploration. The open question is whether the universe will offer, or we ourselves can create, ever more complex and sophisticated opportunities to delight and challenge ever more powerful minds.
Relation to Friendly AI
An artificial general intelligence which is created to help humanity (Friendly AI), and which grows to be much more powerful than us, must have as its goal the promotion of the human value system.
A: Fun theory

Q: Futility of chaos
A: A complex of related ideas having to do with the impossibility of generating useful work from entropy — a position which holds against the ideas that e.g:

Q: A complex of related ideas having to do with the impossibility of generating useful work from entropy — a position which holds against the ideas that e.g:
A: Futility of chaos

Q: Future
A: The usual way in which we plan today for tomorrow is in yesterday's vocabulary. We do so, because we try to get away with the concepts we are familiar with and that have acquired their meanings in our past experience. Of course, the words and the concepts don't quite fit because our future differs from our past, but then we stretch them a little bit. Linguists are quite familiar with the phenomenon that the meanings of words evolve over time, but also know that this is a slow and gradual process.
—E.W. Dijkstra, On the cruelty of really teaching computer science
If it hasn't happened yet but is going to, then it's part of the future. Checking whether or not something is going to happen is notoriously difficult. Luckily, the field of heuristics and biases has given us some insights into what can go wrong. Namely, one problem is that the future elicits far mode, which isn't about truth-seeking or gritty details.

Q: The usual way in which we plan today for tomorrow is in yesterday's vocabulary. We do so, because we try to get away with the concepts we are familiar with and that have acquired their meanings in our past experience. Of course, the words and the concepts don't quite fit because our [...] differs from our past, but then we stretch them a little bit. Linguists are quite familiar with the phenomenon that the meanings of words evolve over time, but also know that this is a slow and gradual process.
—E.W. Dijkstra, On the cruelty of really teaching computer science
If it hasn't happened yet but is going to, then it's part of the [...]. Checking whether or not something is going to happen is notoriously difficult. Luckily, the field of heuristics and biases has given us some insights into what can go wrong. Namely, one problem is that the [...] elicits far mode, which isn't about truth-seeking or gritty details.
A: Future

Q: Fuzzies
A: A hypothetical measurement unit for "warm fuzzy feeling" one gets from believing that one has done good. Unlike utils, fuzzies can be earned through psychological tricks without regard for efficiency. For this reason, it may be a good idea to separate the concerns for actually doing good, for which one might need to shut up and multiply, and for earning fuzzies, to get psychological comfort.

Q: A hypothetical measurement unit for "warm fuzzy feeling" one gets from believing that one has done good. Unlike utils, [...] can be earned through psychological tricks without regard for efficiency. For this reason, it may be a good idea to separate the concerns for actually doing good, for which one might need to shut up and multiply, and for earning [...], to get psychological comfort.
A: Fuzzies

Q: Game theory
A: Game theory attempts to mathematically capture behavior in strategic situations, in which an individual's success in making choices depends on the choices of others.

Q: [...] attempts to mathematically capture behavior in strategic situations, in which an individual's success in making choices depends on the choices of others.
A: Game theory

Q: General knowledge
A: Interdisciplinary, generally applicable knowledge is rarely taught explicitly. Yet it's important to have at least basic knowledge of many areas (as opposed to deep narrowly specialized knowledge), and to apply it to thinking about everything.

Q: Interdisciplinary, generally applicable knowledge is rarely taught explicitly. Yet it's important to have at least basic knowledge of many areas (as opposed to deep narrowly specialized knowledge), and to apply it to thinking about everything.
A: General knowledge

Q: Generalization from fictional evidence
A: The logical fallacy of generalization from fictional evidence consists in drawing real-world conclusions based on statements invented and selected for the purpose of writing fiction.

Q: The logical fallacy of [...] consists in drawing real-world conclusions based on statements invented and selected for the purpose of writing fiction.
A: Generalization from fictional evidence

Q: Giant cheesecake fallacy
A: One often hears, in futurism, a line of reasoning that goes something like this. Someone says: "When technology advances far enough, we’ll be able to build minds far surpassing human intelligence. Now it’s clear, that if you’re baking a cheesecake, how large a cheesecake you can bake depends on your intelligence. A superintelligence could build enormous cheesecakes - cheesecakes the size of cities. And Moore's Law keeps dropping the cost of computing power. By golly, the future will be full of giant cheesecakes!" I call this the Giant Cheesecake Fallacy. It happens whenever the argument leaps directly from capability to actuality, without considering the necessary intermediate of motive.
Here are two examples of reasoning that include a Giant Cheesecake Fallacy:
A sufficiently powerful Artificial Intelligence could overwhelm any human resistance and wipe out humanity. (And the AI would decide to do so.) Therefore we should not build AI.
Or: A sufficiently powerful AI could develop new medical technologies capable of saving millions of human lives. (And the AI would decide to do so.) Therefore we should build AI.
And the natural mistake, once you understand the Giant Cheesecake Fallacy, is to ask: "What will an Artificial Intelligence want?" When trying to talk about Artificial Intelligence, it becomes extremely important to remember that we cannot make any general statement about Artificial Intelligences because the design space is too large. People talk about "AIs" as if all AIs formed a single tribe, an ethnic stereotype. Now, it might make sense to talk about "the human species" as a natural category, because we humans all have essentially the same brain architecture - limbic system, cerebellum, visual cortex, prefrontal cortex, and so on. But the term “Artificial Intelligence” refers to a vastly larger space of possibilities than this. When we talk about “AIs” we are really talking about minds-in-general. Imagine a map of mind design space. In one corner, a tiny little circle contains all humans. And then all the rest of that huge sphere is the space of minds in general.

Q: One often hears, in futurism, a line of reasoning that goes something like this. Someone says: "When technology advances far enough, we’ll be able to build minds far surpassing human intelligence. Now it’s clear, that if you’re baking a cheesecake, how large a cheesecake you can bake depends on your intelligence. A superintelligence could build enormous cheesecakes - cheesecakes the size of cities. And Moore's Law keeps dropping the cost of computing power. By golly, the future will be full of giant cheesecakes!" I call this the Giant Cheesecake Fallacy. It happens whenever the argument leaps directly from capability to actuality, without considering the necessary intermediate of motive.
Here are two examples of reasoning that include a Giant Cheesecake Fallacy:
A sufficiently powerful Artificial Intelligence could overwhelm any human resistance and wipe out humanity. (And the AI would decide to do so.) Therefore we should not build AI.
Or: A sufficiently powerful AI could develop new medical technologies capable of saving millions of human lives. (And the AI would decide to do so.) Therefore we should build AI.
And the natural mistake, once you understand the Giant Cheesecake Fallacy, is to ask: "What will an Artificial Intelligence want?" When trying to talk about Artificial Intelligence, it becomes extremely important to remember that we cannot make any general statement about Artificial Intelligences because the design space is too large. People talk about "AIs" as if all AIs formed a single tribe, an ethnic stereotype. Now, it might make sense to talk about "the human species" as a natural category, because we humans all have essentially the same brain architecture - limbic system, cerebellum, visual cortex, prefrontal cortex, and so on. But the term “Artificial Intelligence” refers to a vastly larger space of possibilities than this. When we talk about “AIs” we are really talking about minds-in-general. Imagine a map of mind design space. In one corner, a tiny little circle contains all humans. And then all the rest of that huge sphere is the space of minds in general.
A: Giant cheesecake fallacy

Q: Goodhart\'s law
A: Goodhart's law states that once a certain indicator of success is made a target of a social or economic policy, it will lose the information content that would qualify it to play such a role.

Q: [...] states that once a certain indicator of success is made a target of a social or economic policy, it will lose the information content that would qualify it to play such a role.
A: Goodhart\'s law

Q: Great Filter
A: The Great Filter is a very improbable step leading to an interstellar civilization, its existence is a proposed explanation for the Fermi Paradox. Robin Hanson coined the term in his 1998 essay The Great Filter - Are We Almost Past It?.
Motivation
If we know that many other habitable planets - fulfilling the conditions for life to flourish until it develops into a interstellar civilization - are bound to exist and at the same time know about the lack of evidence for the existence of other civilizations this implies the existence of a extremely improbable step, a Great Filter, in the path leading from the basics conditions for originating life to the emergence of interstellar civilizations.
Should we worry?
The Great Filter might be the transition from prokaryotes to eukaryotes, or from unicellular to multicellular life forms, or yet the appearance of photosynthesis. Should this be the case, we have already passed through the improbable region, the filter is behind us and we shouldn’t worry. But the hard step might also be ahead of us: to survive the creation of nuclear bombs, AGI biotechnology, nanotechnology or an asteroid impact [1]. In this scenario, we are passing or will pass through the improbable region and we should worry.  Estimating the location of the Great Filter is a crucial task for dealing with Existential risk. Many efforts have been made in that direction, but much remains uncertain.
Estimations
Traces of life in other planets are an evidence for a later Great Filter[2]. Any indication, anywhere in the universe, that life evolved until a period anterior to ours it’s an indication that evolving until such anterior period isn’t so improbable.  Hence, the Great Filter probable location shifts in the future direction, enhancing the chances he is just ahead of us.
The study of past mass extinctions and astrobiology can provide ideas for the estimation of the Great Filter probabilistic distribution. However, there are many difficulties involved. For instance, one can not estimate the improbability of a step based on its completion time. Robin Hanson gives the following example in his seminal paper:
“…say you have one hour to pick five locks by trial and error, locks with 1,2,3,4, and 5 dials of ten numbers, so that the expected time to pick each lock is .01,.1, 1, 10, and 100 hours respectively. Then just looking at those rare cases when you do pick all five locks in the hour, the average time to pick the first two locks would be .0096 and .075 hours respectively, close to the usual expected times of .01 and .1 hours. The average time to pick the third lock, however, would be .20 hours, and the average time for the other two locks, and the average time left over at the end, would be .24 hours. That is, conditional on success, all the hard steps, no matter how hard, take about the same time, while easy steps take about their usual time”
Consequences
On a subsequent paper, Hanson constructs a simulation of the hard steps distribution given earth habitable time, and concludes that there is a total of 4 to 7 hard steps uniformly distributed . It also shows that since hominid evolved there has been at least one hard step and that the best extinction model that fits all these requirements is William Schopf’ model. Taking evolutionary arguments for AGI and observational selection effects together, Bostrom and Shulman argue that Hanson’s results can narrow the bounds on where AGI engineering difficulty can be found.

Q: The [...] is a very improbable step leading to an interstellar civilization, its existence is a proposed explanation for the Fermi Paradox. Robin Hanson coined the term in his 1998 essay The [...] - Are We Almost Past It?.
Motivation
If we know that many other habitable planets - fulfilling the conditions for life to flourish until it develops into a interstellar civilization - are bound to exist and at the same time know about the lack of evidence for the existence of other civilizations this implies the existence of a extremely improbable step, a [...], in the path leading from the basics conditions for originating life to the emergence of interstellar civilizations.
Should we worry?
The [...] might be the transition from prokaryotes to eukaryotes, or from unicellular to multicellular life forms, or yet the appearance of photosynthesis. Should this be the case, we have already passed through the improbable region, the filter is behind us and we shouldn’t worry. But the hard step might also be ahead of us: to survive the creation of nuclear bombs, AGI biotechnology, nanotechnology or an asteroid impact [1]. In this scenario, we are passing or will pass through the improbable region and we should worry.  Estimating the location of the [...] is a crucial task for dealing with Existential risk. Many efforts have been made in that direction, but much remains uncertain.
Estimations
Traces of life in other planets are an evidence for a later [...][2]. Any indication, anywhere in the universe, that life evolved until a period anterior to ours it’s an indication that evolving until such anterior period isn’t so improbable.  Hence, the [...] probable location shifts in the future direction, enhancing the chances he is just ahead of us.
The study of past mass extinctions and astrobiology can provide ideas for the estimation of the [...] probabilistic distribution. However, there are many difficulties involved. For instance, one can not estimate the improbability of a step based on its completion time. Robin Hanson gives the following example in his seminal paper:
“…say you have one hour to pick five locks by trial and error, locks with 1,2,3,4, and 5 dials of ten numbers, so that the expected time to pick each lock is .01,.1, 1, 10, and 100 hours respectively. Then just looking at those rare cases when you do pick all five locks in the hour, the average time to pick the first two locks would be .0096 and .075 hours respectively, close to the usual expected times of .01 and .1 hours. The average time to pick the third lock, however, would be .20 hours, and the average time for the other two locks, and the average time left over at the end, would be .24 hours. That is, conditional on success, all the hard steps, no matter how hard, take about the same time, while easy steps take about their usual time”
Consequences
On a subsequent paper, Hanson constructs a simulation of the hard steps distribution given earth habitable time, and concludes that there is a total of 4 to 7 hard steps uniformly distributed . It also shows that since hominid evolved there has been at least one hard step and that the best extinction model that fits all these requirements is William Schopf’ model. Taking evolutionary arguments for AGI and observational selection effects together, Bostrom and Shulman argue that Hanson’s results can narrow the bounds on where AGI engineering difficulty can be found.
A: Great Filter

Q: Group rationality
A: In almost anything, individuals are inferior to groups. Several articles address this concern regarding rationality.

Q: In almost anything, individuals are inferior to groups. Several articles address this concern regarding rationality.
A: Group rationality

Q: Group selection
A: People who are unfamiliar with evolutionary theory sometimes propose that a feature of the organism is there for the good of the group - for example, that human religion is an adaptation to make human groups more cohesive, since religious groups outfight nonreligious groups.
Postulating group selection is guaranteed to make professional evolutionary biologists roll up their eyes and sigh.
Consider two groups on different sides of a mountain: in group A, each mother gives birth to 2 males and 2 females; in group B, each mother gives birth to 3 females and 1 male.  Group A and group B will have the same number of children, but group B will have 50% more grandchildren and 125% more great-grandchildren.
But consider:  The rarer males become, the more reproductively valuable they become - not to the group, but to the individual parent.  If all the females are doing what's good for the group and birthing 1 male per 10 females, then you can make a genetic killing by birthing all males, each of whom will have (on average) ten times as many grandchildren as their female cousins.
So while group selection ought to favor more girls, individual selection favors equal investment in male and female offspring.  Just by looking at the statistics of a maternity ward, you can see that the quantitative balance between group selection forces and individual selection forces is overwhelmingly tilted in favor of individual selection in Homo sapiens.
Group selection is extremely hard to make work mathematically.  In this simulation, for example, the cost to altruists is 3% of fitness, pure altruist groups have a fitness twice as great as pure selfish groups, the subpopulation size is 25, and 20% of all deaths are replaced with messengers from another group.  The result is polymorphic for selfishness and altruism.  If the subpopulation size is doubled to 50, selfishness is fixed.  If the cost to altruists is increased to 6%, selfishness is fixed.  If the altruistic benefit is decreased by half, selfishness is fixed or in large majority.  Neighborhood-groups must be very small, with only around 5 members, for group selection to operate when the cost of altruism exceeds 10%.
To the best of this editor's knowledge, no definite example of a group-level adaptation has ever been observed in a mammalian species.  Ever.
Hence, postulating group selection in any species - let alone in humans - is guaranteed to make professional evolutionary biologists roll their eyes.
It seems to be extremely popular among a certain sort of amateur evolutionary theorist, though - there's a certain sort of person who, if they don't know about the incredible mathematical difficulty, will find it very satisfying to speculate about adaptations for the good of the group.
The historical fiasco of group selectionism is relied on as a (clear-cut) case in point of the dangers of anthropomorphism.

Q: People who are unfamiliar with evolutionary theory sometimes propose that a feature of the organism is there for the good of the group - for example, that human religion is an adaptation to make human groups more cohesive, since religious groups outfight nonreligious groups.
Postulating [...] is guaranteed to make professional evolutionary biologists roll up their eyes and sigh.
Consider two groups on different sides of a mountain: in group A, each mother gives birth to 2 males and 2 females; in group B, each mother gives birth to 3 females and 1 male.  Group A and group B will have the same number of children, but group B will have 50% more grandchildren and 125% more great-grandchildren.
But consider:  The rarer males become, the more reproductively valuable they become - not to the group, but to the individual parent.  If all the females are doing what's good for the group and birthing 1 male per 10 females, then you can make a genetic killing by birthing all males, each of whom will have (on average) ten times as many grandchildren as their female cousins.
So while [...] ought to favor more girls, individual selection favors equal investment in male and female offspring.  Just by looking at the statistics of a maternity ward, you can see that the quantitative balance between [...] forces and individual selection forces is overwhelmingly tilted in favor of individual selection in Homo sapiens.
[...] is extremely hard to make work mathematically.  In this simulation, for example, the cost to altruists is 3% of fitness, pure altruist groups have a fitness twice as great as pure selfish groups, the subpopulation size is 25, and 20% of all deaths are replaced with messengers from another group.  The result is polymorphic for selfishness and altruism.  If the subpopulation size is doubled to 50, selfishness is fixed.  If the cost to altruists is increased to 6%, selfishness is fixed.  If the altruistic benefit is decreased by half, selfishness is fixed or in large majority.  Neighborhood-groups must be very small, with only around 5 members, for [...] to operate when the cost of altruism exceeds 10%.
To the best of this editor's knowledge, no definite example of a group-level adaptation has ever been observed in a mammalian species.  Ever.
Hence, postulating [...] in any species - let alone in humans - is guaranteed to make professional evolutionary biologists roll their eyes.
It seems to be extremely popular among a certain sort of amateur evolutionary theorist, though - there's a certain sort of person who, if they don't know about the incredible mathematical difficulty, will find it very satisfying to speculate about adaptations for the good of the group.
The historical fiasco of [...]ism is relied on as a (clear-cut) case in point of the dangers of anthropomorphism.
A: Group selection

Q: Groupthink
A: Groupthink is a very well-documented source of bias in cognitive psychology.  It refers to the tendency of humans to tend to agree with each other, and hold back objections or dissent even when the group is wrong.
There seems to be a balance of tensions between evaporative cooling of beliefs leading to groupthink, and extremely biased color politics.
Contents
1 Echo chamber
2 "Virtual community"
3 Blog posts
4 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
  Echo chamber
An echo chamber is a group of media sources that parrot each other by constantly and unquestioningly reporting a single, biased point of view, thus creating the illusion of consensus.
  "Virtual community"
In computer-mediated deliberation, the ideology of so-called "virtual community" has been implicated as a damaging source of groupthink.  In the real world, a "community" is an ethical and political compromise of values due to a shared need for protection from bodily harm or harm to one's surrounding environment.  A "virtual community" has no such natural values but still retains the mind-killing illusion of protection, and is thus liable to turn into a cult.

Q: [...] is a very well-documented source of bias in cognitive psychology.  It refers to the tendency of humans to tend to agree with each other, and hold back objections or dissent even when the group is wrong.
There seems to be a balance of tensions between evaporative cooling of beliefs leading to [...], and extremely biased color politics.
Contents
1 Echo chamber
2 "Virtual community"
3 Blog posts
4 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
  Echo chamber
An echo chamber is a group of media sources that parrot each other by constantly and unquestioningly reporting a single, biased point of view, thus creating the illusion of consensus.
  "Virtual community"
In computer-mediated deliberation, the ideology of so-called "virtual community" has been implicated as a damaging source of [...].  In the real world, a "community" is an ethical and political compromise of values due to a shared need for protection from bodily harm or harm to one's surrounding environment.  A "virtual community" has no such natural values but still retains the mind-killing illusion of protection, and is thus liable to turn into a cult.
A: Groupthink

Q: Guessing the teacher\'s password
A: Proper beliefs pay rent by functioning as anticipation-controllers; a proper belief is not just a string of words, but a cognitive structure fully integrated with the believer's model of how the world simply is. Unfortunately, many students in traditional schools become accustomed to guessing the teacher's password, repeating back words that make it superficially sound as if they understand the topic, when in fact they don't.

Q: Proper beliefs pay rent by functioning as anticipation-controllers; a proper belief is not just a string of words, but a cognitive structure fully integrated with the believer's model of how the world simply is. Unfortunately, many students in traditional schools become accustomed to [...], repeating back words that make it superficially sound as if they understand the topic, when in fact they don't.
A: Guessing the teacher\'s password

Q: Halo effect
A: Let's say that you know someone who not only seems very intelligent, but also honest, altruistic, kindly, and serene.  You should be suspicious that some of these perceived characteristics are influencing your perception of the others.  Maybe the person is genuinely intelligent, honest, and altruistic, but not all that kindly or serene.  You should be suspicious if the people you know seem to separate too cleanly into devils and angels.

Q: Let's say that you know someone who not only seems very intelligent, but also honest, altruistic, kindly, and serene.  You should be suspicious that some of these perceived characteristics are influencing your perception of the others.  Maybe the person is genuinely intelligent, honest, and altruistic, but not all that kindly or serene.  You should be suspicious if the people you know seem to separate too cleanly into devils and angels.
A: Halo effect

Q: AI takeoff
A: AI takeoff refers to a point in the future where Artificial General Intelligence recursively self-improves. This will lead to an increase in intelligence, and will likely lead to an increase in computing power and other resources. The speed at which a AGI may expand is usually split into “soft” and “hard” takeoff scenarios.
Contents
1 Soft takeoff
2 Hard takeoff
3 Blog posts
4 External links
5 See Also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Soft takeoff
A soft takeoff refers to an SAI that would self-improve over a period of months, years or decades. This could be due to either the learning algorithm being too demanding for the hardware or because the AI relies on experiencing feedback from the real-world that would have to be played out in real time. By maintaining control of the AGI’s ascent it should be easier for a Friendly AI to emerge.
Vernor Vinge, Hans Moravec and Ray Kurzweil have all expressed the view that soft takeoff is  preferable to a hard takeoff as it would be both safer and easier to engineer.
Hard takeoff
A hard takeoff refers to AGI expansion in a matter of minutes, hours or days. This scenario is widely considered much more precarious, as this involves an AGI rapidly ascending in power without human control. This may result in unexpected or undesired behavior (i.e. Unfriendly AI). A hard takeoff can either be defined as a system with vastly greater intelligence or one that has acquired extensive computing resources (e.g. control of the Internet).
The feasibility of “hard takeoff” has been addressed by Hugo de Garis, Eliezer Yudkowsky, Ben Goertzel, Nick Bostrom and Michael Anissimov. However, it is widely agreed that a hard takeoff is something to be avoided due to the risks.

Q: [...] refers to a point in the future where Artificial General Intelligence recursively self-improves. This will lead to an increase in intelligence, and will likely lead to an increase in computing power and other resources. The speed at which a AGI may expand is usually split into “soft” and “hard” takeoff scenarios.
Contents
1 Soft takeoff
2 Hard takeoff
3 Blog posts
4 External links
5 See Also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Soft takeoff
A soft takeoff refers to an SAI that would self-improve over a period of months, years or decades. This could be due to either the learning algorithm being too demanding for the hardware or because the AI relies on experiencing feedback from the real-world that would have to be played out in real time. By maintaining control of the AGI’s ascent it should be easier for a Friendly AI to emerge.
Vernor Vinge, Hans Moravec and Ray Kurzweil have all expressed the view that soft takeoff is  preferable to a hard takeoff as it would be both safer and easier to engineer.
Hard takeoff
A hard takeoff refers to AGI expansion in a matter of minutes, hours or days. This scenario is widely considered much more precarious, as this involves an AGI rapidly ascending in power without human control. This may result in unexpected or undesired behavior (i.e. Unfriendly AI). A hard takeoff can either be defined as a system with vastly greater intelligence or one that has acquired extensive computing resources (e.g. control of the Internet).
The feasibility of “hard takeoff” has been addressed by Hugo de Garis, Eliezer Yudkowsky, Ben Goertzel, Nick Bostrom and Michael Anissimov. However, it is widely agreed that a hard takeoff is something to be avoided due to the risks.
A: AI takeoff

Q: Hedon
A: The unit philosophers use to quantify pleasure. Hedons are abstract units with no standardized metric: the assignment of hedons to experiences is typically very broad, loose, subjective, and case dependent. Negative hedons can stand for whatever detracts from a pleasurable experience, or for pain -- though sometimes the unit used for pain is the "dolor".
For example, we might say that eating an ice cream cone yields 3 hedons to a normal individual, while solving an intensely difficult logic puzzle yields 15 hedons. The frustration undergone in the course of figuring out the puzzle might be judged as -20 hedons (or 20 dolors), depending on the puzzle's difficulty and the individual's temperament. The specific numbers chosen are generally not too important; the main point is to give a rough sketch of how the enjoyment of one experience relates to others.
Utilitarians often employ hedons to compare the expected net outcome of pleasure for a given scenario. Hedonism is an ethical theory that sees the maximization of hedons (and the minimization of dolors) as the most desirable good.

Q: The unit philosophers use to quantify pleasure. [...]s are abstract units with no standardized metric: the assignment of [...]s to experiences is typically very broad, loose, subjective, and case dependent. Negative [...]s can stand for whatever detracts from a pleasurable experience, or for pain -- though sometimes the unit used for pain is the "dolor".
For example, we might say that eating an ice cream cone yields 3 [...]s to a normal individual, while solving an intensely difficult logic puzzle yields 15 [...]s. The frustration undergone in the course of figuring out the puzzle might be judged as -20 [...]s (or 20 dolors), depending on the puzzle's difficulty and the individual's temperament. The specific numbers chosen are generally not too important; the main point is to give a rough sketch of how the enjoyment of one experience relates to others.
Utilitarians often employ [...]s to compare the expected net outcome of pleasure for a given scenario. [...]ism is an ethical theory that sees the maximization of [...]s (and the minimization of dolors) as the most desirable good.
A: Hedon

Q: Hedonism
A: The term hedonism refers to a set of philosophies which hold that the highest goal is to maximize pleasure, or more precisely pleasure minus pain.

Q: The term [...] refers to a set of philosophies which hold that the highest goal is to maximize pleasure, or more precisely pleasure minus pain.
A: Hedonism

Q: Heuristic
A: A heuristic is a quick, intuitive strategy for reasoning or decision making, as opposed to more formal methods. Heuristics require much less time and energy to use, but sometimes go awry, producing bias. Common heuristics include:

Q: A [...] is a quick, intuitive strategy for reasoning or decision making, as opposed to more formal methods. [...]s require much less time and energy to use, but sometimes go awry, producing bias. Common [...]s include:
A: Heuristic

Q: Heuristics and biases
A: The heuristics and biases program in cognitive psychology tries to work backward from biases (experimentally reproducible human errors) to heuristics (the underlying mechanisms at work in the brain).

Q: The [...] program in cognitive psychology tries to work backward from biases (experimentally reproducible human errors) to heuristics (the underlying mechanisms at work in the brain).
A: Heuristics and biases

Q: Hindsight bias
A: Hindsight bias is a tendency to overestimate the foreseeability of events that have actually happened.  I.e., subjects given information about X, and asked to assign a probability that X will happen, assign much lower probabilities than subjects who are given the same information about X, are told that X actually happened, and asked to estimate the foreseeable probability of X.  Experiments also show that instructing subjects to "avoid hindsight bias" has little or no effect.

Q: [...] is a tendency to overestimate the foreseeability of events that have actually happened.  I.e., subjects given information about X, and asked to assign a probability that X will happen, assign much lower probabilities than subjects who are given the same information about X, are told that X actually happened, and asked to estimate the foreseeable probability of X.  Experiments also show that instructing subjects to "avoid [...]" has little or no effect.
A: Hindsight bias

Q: Hollywood rationality
A: Hollywood rationality is a popular stereotype of what rationality is about. It presents "rationalists" as word-obsessed, floating in endless verbal space disconnected from reality, reciting figures to more decimal places than are necessary, or speaking in a dull monotone. Hollywood rationalists also tend to either not have strong (or any) emotions. When they do, they often do not express them. Needless to say, this has nothing to do with actual rationality.

Q: [...] is a popular stereotype of what rationality is about. It presents "rationalists" as word-obsessed, floating in endless verbal space disconnected from reality, reciting figures to more decimal places than are necessary, or speaking in a dull monotone. Hollywood rationalists also tend to either not have strong (or any) emotions. When they do, they often do not express them. Needless to say, this has nothing to do with actual rationality.
A: Hollywood rationality

Q: Hope
A: Persisting in clutching to a hope may be disastrous. Be ready to admit you lost, update on the data that says you did.

Q: Persisting in clutching to a [...] may be disastrous. Be ready to admit you lost, update on the data that says you did.
A: Hope

Q: How an algorithm feels
A: Our philosophical intuitions do not rain down on us as manna from heaven; they are generated by algorithms in the human brain. Our philosophical intuitions, indeed, are how these particular cognitive algorithms feel from the inside.
To dissolve a philosophical dilemma, it often suffices to understand the cognitive algorithm that generates the appearance of the dilemma - if you understand the algorithm in sufficient detail. It is not enough to say "An algorithm does it!" - this might as well be magic. It takes a detailed step-by-step walkthrough.
Michael Vassar has observed that conventional philosophers seem to be spectacularly bad at understanding that their intuitions are generated by cognitive algorithms. This may be why works of serious reductionism get written by Artificial Intelligence people instead of conventional philosophers.

Q: Our philosophical intuitions do not rain down on us as manna from heaven; they are generated by algorithms in the human brain. Our philosophical intuitions, indeed, are how these particular cognitive algorithms feel from the inside.
To dissolve a philosophical dilemma, it often suffices to understand the cognitive algorithm that generates the appearance of the dilemma - if you understand the algorithm in sufficient detail. It is not enough to say "An algorithm does it!" - this might as well be magic. It takes a detailed step-by-step walkthrough.
Michael Vassar has observed that conventional philosophers seem to be spectacularly bad at understanding that their intuitions are generated by cognitive algorithms. This may be why works of serious reductionism get written by Artificial Intelligence people instead of conventional philosophers.
A: How an algorithm feels

Q: How To Actually Change Your Mind
A: A sequence on the ultra-high-level penultimate technique of rationality:  How to actually change your mind.
Posts in this collection are scattered across widely ranging dates and times.
Contents
1 Subsequences
1.1 Politics is the Mind-Killer
1.2 Death Spirals and the Cult Attractor
1.3 Seeing with Fresh Eyes
1.4 Noticing Confusion
1.5 Against Rationalization
1.6 Against Doublethink
1.7 Overly Convenient Excuses
1.8 Letting Go
2 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Subsequences
Politics is the Mind-Killer
Most important posts:
A Fable of Science and Politics
Politics is the Mind-Killer
Policy Debates Should Not Appear One-Sided
The Scales of Justice, the Notebook of Rationality
Reversed Stupidity is Not Intelligence
Argument Screens Off Authority
Hug the Query
Death Spirals and the Cult Attractor
Most important posts:
The Affect Heuristic
The Halo Effect
Affective Death Spirals
Resist the Happy Death Spiral
Uncritical Supercriticality
Seeing with Fresh Eyes
Anchoring and Adjustment
Priming and Contamination
Do We Believe Everything We're Told?
Cached Thoughts
The 'Outside the Box' Box
Original Seeing
The Logical Fallacy of Generalization from Fictional Evidence
How to Seem (and Be) Deep
We Change Our Minds Less Often Than We Think
Hold Off On Proposing Solutions
On Expressing Your Concerns
The Genetic Fallacy
Noticing Confusion
(Heavy overlap with Mysterious Answers to Mysterious Questions.)

Q: A sequence on the ultra-high-level penultimate technique of rationality:  How to actually change your mind.
Posts in this collection are scattered across widely ranging dates and times.
Contents
1 Subsequences
1.1 Politics is the Mind-Killer
1.2 Death Spirals and the Cult Attractor
1.3 Seeing with Fresh Eyes
1.4 Noticing Confusion
1.5 Against Rationalization
1.6 Against Doublethink
1.7 Overly Convenient Excuses
1.8 Letting Go
2 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Subsequences
Politics is the Mind-Killer
Most important posts:
A Fable of Science and Politics
Politics is the Mind-Killer
Policy Debates Should Not Appear One-Sided
The Scales of Justice, the Notebook of Rationality
Reversed Stupidity is Not Intelligence
Argument Screens Off Authority
Hug the Query
Death Spirals and the Cult Attractor
Most important posts:
The Affect Heuristic
The Halo Effect
Affective Death Spirals
Resist the Happy Death Spiral
Uncritical Supercriticality
Seeing with Fresh Eyes
Anchoring and Adjustment
Priming and Contamination
Do We Believe Everything We're Told?
Cached Thoughts
The 'Outside the Box' Box
Original Seeing
The Logical Fallacy of Generalization from Fictional Evidence
How to Seem (and Be) Deep
We Change Our Minds Less Often Than We Think
Hold Off On Proposing Solutions
On Expressing Your Concerns
The Genetic Fallacy
Noticing Confusion
(Heavy overlap with Mysterious Answers to Mysterious Questions.)
A: How To Actually Change Your Mind

Q: Human universal
A: Complex adaptations within a sexually reproducing species need all of their parts, or almost all of their parts, to be universal within the gene pool.
Let's say that you have a complex adaptation with six interdependent parts, and that each of the six genes is independently at ten percent frequency in the population.  The chance of assembling a whole working adaptation is literally a million to one; and the average fitness of the genes is tiny, and they will not increase in frequency.
One bird may have slightly smoother feathers than another, but they will both have wings.  A single mutation can be possessed by some lucky members of a species, and not by others - but single mutations don't correspond to the sort of complex, powerful machinery that underlies the potency of biology. By the time an adaptation gets to be really sophisticated with dozens of genes supporting its highly refined activity, every member of the species has some version of it - barring single mutations that knock out the whole complex.
Applying this logic to human brains in particular, we arrive at the explanation for what is called the psychic unity of mankind.  (Though Eliezer Yudkowsky has used the phrase "psychological unity of humankind" instead.)  In every known culture, humans seem to experience joy, sadness, fear, disgust, anger, and surprise.  In every known culture, these emotions are indicated by the same facial expressions.  (Citation needed to Paul Ekman.)
Donald E. Brown has compiled a list of over a hundred human universals - traits found in every culture ever studied, most of them so universal that anthropologists don't even bother to note them explicitly.

Q: Complex adaptations within a sexually reproducing species need all of their parts, or almost all of their parts, to be universal within the gene pool.
Let's say that you have a complex adaptation with six interdependent parts, and that each of the six genes is independently at ten percent frequency in the population.  The chance of assembling a whole working adaptation is literally a million to one; and the average fitness of the genes is tiny, and they will not increase in frequency.
One bird may have slightly smoother feathers than another, but they will both have wings.  A single mutation can be possessed by some lucky members of a species, and not by others - but single mutations don't correspond to the sort of complex, powerful machinery that underlies the potency of biology. By the time an adaptation gets to be really sophisticated with dozens of genes supporting its highly refined activity, every member of the species has some version of it - barring single mutations that knock out the whole complex.
Applying this logic to human brains in particular, we arrive at the explanation for what is called the psychic unity of mankind.  (Though Eliezer Yudkowsky has used the phrase "psychological unity of humankind" instead.)  In every known culture, humans seem to experience joy, sadness, fear, disgust, anger, and surprise.  In every known culture, these emotions are indicated by the same facial expressions.  (Citation needed to Paul Ekman.)
Donald E. Brown has compiled a list of over a hundred [...]s - traits found in every culture ever studied, most of them so universal that anthropologists don't even bother to note them explicitly.
A: Human universal

Q: Humility
A: To be humble is to take specific actions in anticipation of your own errors.  To confess your fallibility and then do nothing about it is not humble; it is boasting of your modesty.
—Twelve Virtues of Rationality
Not to be confused with social modesty, or motivated skepticism (aka disconfirmation bias).

Q: To be humble is to take specific actions in anticipation of your own errors.  To confess your fallibility and then do nothing about it is not humble; it is boasting of your modesty.
—Twelve Virtues of Rationality
Not to be confused with social modesty, or motivated skepticism (aka disconfirmation bias).
A: Humility

Q: Hypocrisy
A: Formally, hypocrisy is the act of claiming to motives, morals and standards one does not possess. Informally, it refers to not living up the the standards that one espouses, whether or not one sincerely believes those standards.

Q: Formally, [...] is the act of claiming to motives, morals and standards one does not possess. Informally, it refers to not living up the the standards that one espouses, whether or not one sincerely believes those standards.
A: Hypocrisy

Q: I don\'t know
A: What we cannot speak about we must pass over in silence.
—Ludwig Wittgenstein, Tractatus Logico-Philosophicus
In order to even phrase a question, you need to know something about a domain. The words "I don't know" might be a convenient shorthand to express to the unenlightened that you don't know very much in particular—that you don't think you can do better than they can, or than your ignorance prior. But in real life, you are constantly making decisions under uncertainty: the null plan is still a plan, refusing to choose is itself a choice, and by your choices, you implicitly take bets at some odds, whether or not you explicitly conceive of yourself as doing so.

Q: What we cannot speak about we must pass over in silence.
—Ludwig Wittgenstein, Tractatus Logico-Philosophicus
In order to even phrase a question, you need to know something about a domain. The words "[...]" might be a convenient shorthand to express to the unenlightened that you don't know very much in particular—that you don't think you can do better than they can, or than your ignorance prior. But in real life, you are constantly making decisions under uncertainty: the null plan is still a plan, refusing to choose is itself a choice, and by your choices, you implicitly take bets at some odds, whether or not you explicitly conceive of yourself as doing so.
A: I don\'t know

Q: Ignorance prior
A: An ignorance prior is a prior probability distribution where equal probability is assigned to all possibilities. For instance, at the beginning of a Monty Hall game, the player has no information about which of the three doors has the car behind it. Thus they should assign 1/3 probability to each.

Q: An [...] is a prior probability distribution where equal probability is assigned to all possibilities. For instance, at the beginning of a Monty Hall game, the player has no information about which of the three doors has the car behind it. Thus they should assign 1/3 probability to each.
A: Ignorance prior

Q: Illusion of transparency
A: The illusion of transparency is the misleading impression that your words convey more to others than they really do. Words are a means of communication, but they don't in themselves contain meaning. The word apple is just five letters, two syllables. I use it to refer to a concept and its associations in my mind, under the reasonable assumption that it refers to a similar concept and group of associations in your mind; this is the only power words have, great though it may be. Unfortunately, it's easy to lose track of this fact, think as if your words have meanings inherently encoded in them, leading to a tendency to systematically overestimate the effectiveness of communication.

Q: The [...] is the misleading impression that your words convey more to others than they really do. Words are a means of communication, but they don't in themselves contain meaning. The word apple is just five letters, two syllables. I use it to refer to a concept and its associations in my mind, under the reasonable assumption that it refers to a similar concept and group of associations in your mind; this is the only power words have, great though it may be. Unfortunately, it's easy to lose track of this fact, think as if your words have meanings inherently encoded in them, leading to a tendency to systematically overestimate the effectiveness of communication.
A: Illusion of transparency

Q: Impossibility
A: Careful use of language dictates that we distinguish between several senses in which something can be said to be impossible. Some things are logically impossible: you can't have a square circle or an object that is both perfectly black and perfectly not-black. Also, in our reductionist universe operating according to universal physical laws, some things are physically impossible based on our model of how things work, even they are not obviously contradictory or contrary to reason: for example, the laws of thermodynamics give us a strong guarantee that there can never be a perpetual motion machine.
It can be tempting to label as impossible very difficult problems which you have no idea how to solve. But the apparent lack of a solution is not a strong guarantee that no solution can exist in the way that the laws of thermodynamics, or Godel's incompleteness results, give us proofs that something can not be accomplished. A blank map does not correspond to a blank territory; in the absence of a proof that a problem is insolvable, you can't be confident that you're not just overlooking something that a greater intelligence would spot in an instant.

Q: Careful use of language dictates that we distinguish between several senses in which something can be said to be impossible. Some things are logically impossible: you can't have a square circle or an object that is both perfectly black and perfectly not-black. Also, in our reductionist universe operating according to universal physical laws, some things are physically impossible based on our model of how things work, even they are not obviously contradictory or contrary to reason: for example, the laws of thermodynamics give us a strong guarantee that there can never be a perpetual motion machine.
It can be tempting to label as impossible very difficult problems which you have no idea how to solve. But the apparent lack of a solution is not a strong guarantee that no solution can exist in the way that the laws of thermodynamics, or Godel's incompleteness results, give us proofs that something can not be accomplished. A blank map does not correspond to a blank territory; in the absence of a proof that a problem is insolvable, you can't be confident that you're not just overlooking something that a greater intelligence would spot in an instant.
A: Impossibility

Q: Improper belief
A: An improper belief is a belief that isn't concerned with describing the territory. A proper belief, on the other hand, requires observations, gets updated upon encountering new evidence, and provides practical benefit in anticipated experience.
Note that the fact that a belief just happens to be true doesn't mean you're right to have it. If you buy a lottery ticket, certain that it's a winning ticket (for no reason), and it happens to be, believing that was still a mistake.
Improper belief is one of the subjects of the Mysterious Answers to Mysterious Questions sequence.  Types of improper belief discussed there include:

Q: An [...] is a belief that isn't concerned with describing the territory. A proper belief, on the other hand, requires observations, gets updated upon encountering new evidence, and provides practical benefit in anticipated experience.
Note that the fact that a belief just happens to be true doesn't mean you're right to have it. If you buy a lottery ticket, certain that it's a winning ticket (for no reason), and it happens to be, believing that was still a mistake.
[...] is one of the subjects of the Mysterious Answers to Mysterious Questions sequence.  Types of [...] discussed there include:
A: Improper belief

Q: In-group bias
A: In-group bias is the preferential treatment of people and ideas associated with your own group. Can lead to color politics, and is a catalyst for affective death spirals.

Q: [...] is the preferential treatment of people and ideas associated with your own group. Can lead to color politics, and is a catalyst for affective death spirals.
A: In-group bias

Q: Incredulity
A: Spending emotional energy on incredulity wastes time you could be using to update.  It repeatedly throws you back into the frame of the old, wrong viewpoint.  It feeds your sense of righteous indignation at reality daring to contradict you.

Q: Spending emotional energy on [...] wastes time you could be using to update.  It repeatedly throws you back into the frame of the old, wrong viewpoint.  It feeds your sense of righteous indignation at reality daring to contradict you.
A: Incredulity

Q: Inductive bias
A: The inductive bias of a learning algorithm is the set of assumptions that the learner uses to predict outputs given inputs that it has not encountered
—Tom Mitchell[1]
Inductive bias refers to your suspicion that if the sun has risen for the last billion days in a row, then it may rise tomorrow as well.  Since it is logically possible that the laws of physics will arbitrarily cease to work and that the sun will not rise tomorrow, coming to this conclusion requires an inductively biased prior.
This sort of bias is not a bad thing - without "inductive bias" you can't draw any conclusion at all from the data.  It's just a different technical meaning attached to the same word.

Q: The [...] of a learning algorithm is the set of assumptions that the learner uses to predict outputs given inputs that it has not encountered
—Tom Mitchell[1]
[...] refers to your suspicion that if the sun has risen for the last billion days in a row, then it may rise tomorrow as well.  Since it is logically possible that the laws of physics will arbitrarily cease to work and that the sun will not rise tomorrow, coming to this conclusion requires an inductively biased prior.
This sort of bias is not a bad thing - without "[...]" you can't draw any conclusion at all from the data.  It's just a different technical meaning attached to the same word.
A: Inductive bias

Q: Inferential distance
A: Inferential distance is a gap between the background knowledge and epistemology of a person trying to explain an idea, and the background knowledge and epistemology of the person trying to understand it.
For example, explaining the evidence for the theory of evolution to a physicist would be easy; even if the physicist didn't already know about evolution, they would understand the concepts of evidence, Occam's razor, naturalistic explanations, and the general orderly nature of the universe. Explaining the evidence for the theory of evolution to someone without a science background would be much harder. Before even mentioning the specific evidence for evolution, you would have to explain the concept of evidence, why some kinds of evidence are more valuable than others, what does and doesn't count as evidence, and so on. This would be unlikely to work during a short conversation.
There is a short inferential distance between you and the physicist; there is a very long inferential distance between you and the person without any science background. Many members of Less Wrong believe that expecting short inferential distances is a classic error. It is also a very difficult problem to solve, since most people will feel offended if you explicitly say that there is too great an inferential distance between you to explain a theory properly. Some people have attempted to explain this through evolutionary psychology: in the ancestral environment, there was minimal difference in knowledge between people, and therefore no need to worry about inferential distances.

Q: [...] is a gap between the background knowledge and epistemology of a person trying to explain an idea, and the background knowledge and epistemology of the person trying to understand it.
For example, explaining the evidence for the theory of evolution to a physicist would be easy; even if the physicist didn't already know about evolution, they would understand the concepts of evidence, Occam's razor, naturalistic explanations, and the general orderly nature of the universe. Explaining the evidence for the theory of evolution to someone without a science background would be much harder. Before even mentioning the specific evidence for evolution, you would have to explain the concept of evidence, why some kinds of evidence are more valuable than others, what does and doesn't count as evidence, and so on. This would be unlikely to work during a short conversation.
There is a short [...] between you and the physicist; there is a very long [...] between you and the person without any science background. Many members of Less Wrong believe that expecting short [...]s is a classic error. It is also a very difficult problem to solve, since most people will feel offended if you explicitly say that there is too great an [...] between you to explain a theory properly. Some people have attempted to explain this through evolutionary psychology: in the ancestral environment, there was minimal difference in knowledge between people, and therefore no need to worry about [...]s.
A: Inferential distance

Q: Information cascade
A: An information cascade occurs when people signal that they have information about something, but actually based their judgment on other people's signals, resulting in a self-reinforcing community opinion that does not necessarily reflect reality.

Q: An [...] occurs when people signal that they have information about something, but actually based their judgment on other people's signals, resulting in a self-reinforcing community opinion that does not necessarily reflect reality.
A: Information cascade

Q: Information hazard
A: An information hazard is a concept coined by Nick Bostrom in a 2011 paper[1] for Review of Contemporary Philosophy. He defines it as follows;
Information hazard: A risk that arises from the dissemination or the potential dissemination of (true) information that may cause harm or enable some agent to cause harm.
Bostrom points out that this is in contrast to the generally accepted principle of information freedom and that, while rare, the possibility of information hazards needs to be considered when making information policies. He proceeds to categorize and define a large number of sub-types of information hazards. For example, he defines artificial intelligence hazard as
Artificial intelligence hazard: There could be computer-related risks in which the threat
would derive primarily from the cognitive sophistication of the program rather than the
specific properties of any actuators to which the system initially has access.
The table below is reproduced from (Bostrom 2011).
TYPOLOGY OF INFORMATION HAZARDS
I. By information transfer mode
Data hazard
Idea hazard
Attention hazard
Template hazard
Signaling hazard
Evocation hazard
II. By effect
   TYPE  SUBTYPE
ADVERSARIAL RISKS
Competiveness hazard
Enemy Hazard
Intellectual property hazard
Commitment hazard
Knowing-too-much hazard
RISKS TO SOCIAL ORGANIZATION AND MARKETS
Norm hazard
Information asymmetry Hazard
Unveiling hazard
Recognition hazard
RISKS OF IRRATIONALITY AND ERROR
Ideological hazard
Distraction and temptation hazard
Role model hazard
Biasing hazard
De-biasing hazard
Neuropsychological hazard
Information-burying hazard
RISKS TO VALUABLE STATES AND ACTIVITIES
Psychological reaction hazard
Disappointment hazard
Spoiler hazard
Mindset hazard
Belief-constituted value hazard
(mixed)  Embarrassment hazard
RISKS FROM INFORMATION TECHNOLOGY SYSTEMS
Information system hazard
Information infrastructure failure hazard
Information infrastructure misuse hazard
Artificial intelligence hazard
RISKS FROM DEVELOPMENT  Development hazard

Q: An [...] is a concept coined by Nick Bostrom in a 2011 paper[1] for Review of Contemporary Philosophy. He defines it as follows;
[...]: A risk that arises from the dissemination or the potential dissemination of (true) information that may cause harm or enable some agent to cause harm.
Bostrom points out that this is in contrast to the generally accepted principle of information freedom and that, while rare, the possibility of [...]s needs to be considered when making information policies. He proceeds to categorize and define a large number of sub-types of [...]s. For example, he defines artificial intelligence hazard as
Artificial intelligence hazard: There could be computer-related risks in which the threat
would derive primarily from the cognitive sophistication of the program rather than the
specific properties of any actuators to which the system initially has access.
The table below is reproduced from (Bostrom 2011).
TYPOLOGY OF INFORMATION HAZARDS
I. By information transfer mode
Data hazard
Idea hazard
Attention hazard
Template hazard
Signaling hazard
Evocation hazard
II. By effect
   TYPE  SUBTYPE
ADVERSARIAL RISKS
Competiveness hazard
Enemy Hazard
Intellectual property hazard
Commitment hazard
Knowing-too-much hazard
RISKS TO SOCIAL ORGANIZATION AND MARKETS
Norm hazard
Information asymmetry Hazard
Unveiling hazard
Recognition hazard
RISKS OF IRRATIONALITY AND ERROR
Ideological hazard
Distraction and temptation hazard
Role model hazard
Biasing hazard
De-biasing hazard
Neuropsychological hazard
Information-burying hazard
RISKS TO VALUABLE STATES AND ACTIVITIES
Psychological reaction hazard
Disappointment hazard
Spoiler hazard
Mindset hazard
Belief-constituted value hazard
(mixed)  Embarrassment hazard
RISKS FROM INFORMATION TECHNOLOGY SYSTEMS
Information system hazard
Information infrastructure failure hazard
Information infrastructure misuse hazard
Artificial intelligence hazard
RISKS FROM DEVELOPMENT  Development hazard
A: Information hazard

Q: Rationality
A: The art of thinking. Systematic methods of attaining a map that reflects the territory, or achieving goals. The primary topic of Less Wrong.
For the canonical article, see What Do We Mean By "Rationality"?.
Contents
1 Epistemic rationality
2 Instrumental rationality
3 Blog posts
4 External links
5 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Epistemic rationality
Believing, and updating on evidence, so as to systematically improve the correspondence between your map and the territory. The art of obtaining beliefs that correspond to reality as closely as possible. This correspondence is commonly termed "truth" or "accuracy", and we're happy to call it that.
Instrumental rationality
Achieving your values. Not necessarily "your values" in the sense of being selfish values or unshared values: "your values" means anything you care about. The art of choosing actions that steer the future toward outcomes ranked higher in your preferences. LW sometimes shortens this to "winning".
For humans, instrumental rationality benefits from deliberation techniques — for identifying your better courses of action — and implementation techinques — to help you act the way you've decided upon.

Q: The art of thinking. Systematic methods of attaining a map that reflects the territory, or achieving goals. The primary topic of Less Wrong.
For the canonical article, see What Do We Mean By "[...]"?.
Contents
1 Epistemic [...]
2 Instrumental [...]
3 Blog posts
4 External links
5 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Epistemic [...]
Believing, and updating on evidence, so as to systematically improve the correspondence between your map and the territory. The art of obtaining beliefs that correspond to reality as closely as possible. This correspondence is commonly termed "truth" or "accuracy", and we're happy to call it that.
Instrumental [...]
Achieving your values. Not necessarily "your values" in the sense of being selfish values or unshared values: "your values" means anything you care about. The art of choosing actions that steer the future toward outcomes ranked higher in your preferences. LW sometimes shortens this to "winning".
For humans, instrumental [...] benefits from deliberation techniques — for identifying your better courses of action — and implementation techinques — to help you act the way you've decided upon.
A: Rationality

Q: Instrumental value
A: An instrumental value is a value pursued for the purpose of achieving other values. Values which are pursued for their own sake are called terminal values.

Q: An [...] is a value pursued for the purpose of achieving other values. Values which are pursued for their own sake are called terminal values.
A: Instrumental value

Q: Intellectual roles
A: Group rationality may be improved when members of the group take on specific intellectual roles. While these roles may be incomplete on their own, each embodies an aspect of proper rationality. If certain roles are biased against, purposefully adopting them might reduce bias.

Q: Group rationality may be improved when members of the group take on specific [...]. While these roles may be incomplete on their own, each embodies an aspect of proper rationality. If certain roles are biased against, purposefully adopting them might reduce bias.
A: Intellectual roles

Q: Joy in discovery
A: The pleasure of figuring things out need not be restricted to professional researchers doing cutting-edge science. The fact that a particular truth has been discovered by someone doesn't mean that particular truth is forever marked with a little XML tag that says <KNOWN>. Knowledge is evidential entanglement between brains and environments; in a disconnected world where it's difficult to explain things, it's hard to see how joy in discovery could feasibly be reserved for one discoverer. The fact that someone else's beliefs are evidentially entangled with their environment doesn't mean that yours can't be, too. Many aspiring rationalists find this (coupled with certain fun-theoretic concerns) a compelling argument for taking joy in every personal discovery, delighting in the truth-finding process without undue concern about what other people already know. Your mileage may vary.

Q: The pleasure of figuring things out need not be restricted to professional researchers doing cutting-edge science. The fact that a particular truth has been discovered by someone doesn't mean that particular truth is forever marked with a little XML tag that says <KNOWN>. Knowledge is evidential entanglement between brains and environments; in a disconnected world where it's difficult to explain things, it's hard to see how [...] could feasibly be reserved for one discoverer. The fact that someone else's beliefs are evidentially entangled with their environment doesn't mean that yours can't be, too. Many aspiring rationalists find this (coupled with certain fun-theoretic concerns) a compelling argument for taking joy in every personal discovery, delighting in the truth-finding process without undue concern about what other people already know. Your mileage may vary.
A: Joy in discovery

Q: Joy in the merely real
A: Alas for those who turn their eyes from zebras and dream of dragons! If we cannot learn to take joy in the merely real, our lives shall be empty indeed.
—A Technical Explanation of Technical Explanation
A phenomenon shouldn't have to be unknowable or irreducible in order to be worth taking joy in, because nothing is inherently mysterious. Any mysterious inexplicability that we percieve is only an artifact of our own ignorance. The "merely real" is all we can ever have.

Q: Alas for those who turn their eyes from zebras and dream of dragons! If we cannot learn to take [...], our lives shall be empty indeed.
—A Technical Explanation of Technical Explanation
A phenomenon shouldn't have to be unknowable or irreducible in order to be worth taking joy in, because nothing is inherently mysterious. Any mysterious inexplicability that we percieve is only an artifact of our own ignorance. The "merely real" is all we can ever have.
A: Joy in the merely real

Q: Lawful intelligence
A: The startling and counterintuitive notion - contradicting both surface appearances and all Deep Wisdom - that intelligence is a manifestation of Order rather than Chaos.  Even creativity and outside-the-box thinking are essentially lawful.
While this is a complete heresy according to the standard religion of Silicon Valley, there are some good mathematical reasons for believing it.

Q: The startling and counterintuitive notion - contradicting both surface appearances and all Deep Wisdom - that intelligence is a manifestation of Order rather than Chaos.  Even creativity and outside-the-box thinking are essentially lawful.
While this is a complete heresy according to the standard religion of Silicon Valley, there are some good mathematical reasons for believing it.
A: Lawful intelligence

Q: Least convenient possible world
A: Least convenient possible world (or LCPW) is a technique for enforcing intellectual honesty, to be used when arguing against an idea. The essence of the technique is to assume that all the specific details will align with the idea against which you are arguing, i.e. to consider the idea in the context of a least convenient possible world, where every circumstance is colluding against your objections and counterarguments. This approach ensures that your objections are strong enough, running minimal risk of being rationalizations for your position.

Q: [...] (or LCPW) is a technique for enforcing intellectual honesty, to be used when arguing against an idea. The essence of the technique is to assume that all the specific details will align with the idea against which you are arguing, i.e. to consider the idea in the context of a [...], where every circumstance is colluding against your objections and counterarguments. This approach ensures that your objections are strong enough, running minimal risk of being rationalizations for your position.
A: Least convenient possible world

Q: Litany of Gendlin
A: What is true is already so.
Owning up to it doesn't make it worse.
Not being open about it doesn't make it go away.
And because it's true, it is what is there to be interacted with.
Anything untrue isn't there to be lived.
People can stand what is true,
for they are already enduring it.

Q: What is true is already so.
Owning up to it doesn't make it worse.
Not being open about it doesn't make it go away.
And because it's true, it is what is there to be interacted with.
Anything untrue isn't there to be lived.
People can stand what is true,
for they are already enduring it.
A: Litany of Gendlin

Q: Litany of Tarski
A: If the box contains a diamond,
I desire to believe that the box contains a diamond;
If the box does not contain a diamond,
I desire to believe that the box does not contain a diamond;
Let me not become attached to beliefs I may not want.
—The Meditation on Curiosity
The Litany of Tarski is actually a litany template that can be stated about any fact. Here's another example:

Q: If the box contains a diamond,
I desire to believe that the box contains a diamond;
If the box does not contain a diamond,
I desire to believe that the box does not contain a diamond;
Let me not become attached to beliefs I may not want.
—The Meditation on Curiosity
The [...] is actually a litany template that can be stated about any fact. Here's another example:
A: Litany of Tarski

Q: Locate the hypothesis
A: When the space of possible answers is large, it takes a large amount of evidence just to think of the correct answer - to promote it to your attention.
Suppose there's a million boxes, and only one of those boxes contains a diamond.  And suppose that we have a locator device, which, waved over a box, always beeps if the box contains a diamond, and beeps with 10% probability if the box does not contain a diamond.  Then a beep is a likelihood ratio of 10:1 in favor of the box containing a diamond, and the box has 1:1,000,000 prior odds of containing the diamond.  If we run the locator over all the boxes, we would get 1 true positive and 100,000 false positives.
When the space of possibilities is large, it takes a  large amount of Bayesian evidence just to locate the truth in hypothesis-space - to raise it to the level of our attention - to select it as one of a manageable number of alternatives to spend time thinking about individually.
In 1919, Sir Arthur Eddington led expeditions to Brazil and to the island of Principe, aiming to observe solar eclipses and thereby test an experimental prediction of Einstein's novel theory of General Relativity.  A journalist asked Einstein what he would do if Eddington's observations failed to match his theory.  Einstein famously replied:  "Then I would feel sorry for the good Lord.  The theory is correct."  This sounds like Einstein is defying the sovereignty of experiment - jumping to conclusions in advance of the experimental data.  But since Einstein was in fact right, it would be extremely improbable for him to have been right just by jumping to conclusions.  Einstein must have had enough rational evidence already in hand to locate General Relativity in theory-space.

Q: When the space of possible answers is large, it takes a large amount of evidence just to think of the correct answer - to promote it to your attention.
Suppose there's a million boxes, and only one of those boxes contains a diamond.  And suppose that we have a locator device, which, waved over a box, always beeps if the box contains a diamond, and beeps with 10% probability if the box does not contain a diamond.  Then a beep is a likelihood ratio of 10:1 in favor of the box containing a diamond, and the box has 1:1,000,000 prior odds of containing the diamond.  If we run the locator over all the boxes, we would get 1 true positive and 100,000 false positives.
When the space of possibilities is large, it takes a  large amount of Bayesian evidence just to locate the truth in hypothesis-space - to raise it to the level of our attention - to select it as one of a manageable number of alternatives to spend time thinking about individually.
In 1919, Sir Arthur Eddington led expeditions to Brazil and to the island of Principe, aiming to observe solar eclipses and thereby test an experimental prediction of Einstein's novel theory of General Relativity.  A journalist asked Einstein what he would do if Eddington's observations failed to match his theory.  Einstein famously replied:  "Then I would feel sorry for the good Lord.  The theory is correct."  This sounds like Einstein is defying the sovereignty of experiment - jumping to conclusions in advance of the experimental data.  But since Einstein was in fact right, it would be extremely improbable for him to have been right just by jumping to conclusions.  Einstein must have had enough rational evidence already in hand to locate General Relativity in theory-space.
A: Locate the hypothesis

Q: Log odds
A: Log odds are an alternate way of expressing probabilities, which simplifies the process of updating them with new evidence. Unfortunately, it is difficult to convert between probability and log odds. The log odds is the log of the odds ratio. Thus, the log odds of A are logit(P(A)) = log(P(A)/P(¬A)).
Thus, in order to find the posterior log odds , one simply adds the prior log odds ratio  by the log of the likelihood ratio .
The base of the logarithm doesn't matter so long as it's consistant, and the natural log is usually used.
Suppose you have a box that has a 5% chance of containing a diamond. You also have a diamond detector that beeps half of the time if there is a diamond, and one fourth of the time if there is not. You wave the diamond detector over the box and it beeps.
The prior log odds of the box containing a diamond are ln(1/19) = -2.94. The log of the likelihood ratio of a beep is ln((1/2)/(1/4)) = ln(2) = 0.69. The posterior log odds are -2.94 + 0.69 = -2.23. This corresponds to an odds ratio of e^-2.23 = 0.108:1, and thus a probability of 0.108/1.108 = 0.097 = 9.7%. The correct answer is actually 9.5%. There was error due to rounding.

Q: [...] are an alternate way of expressing probabilities, which simplifies the process of updating them with new evidence. Unfortunately, it is difficult to convert between probability and [...]. The [...] is the log of the odds ratio. Thus, the [...] of A are logit(P(A)) = log(P(A)/P(¬A)).
Thus, in order to find the posterior [...] , one simply adds the prior [...] ratio  by the log of the likelihood ratio .
The base of the logarithm doesn't matter so long as it's consistant, and the natural log is usually used.
Suppose you have a box that has a 5% chance of containing a diamond. You also have a diamond detector that beeps half of the time if there is a diamond, and one fourth of the time if there is not. You wave the diamond detector over the box and it beeps.
The prior [...] of the box containing a diamond are ln(1/19) = -2.94. The log of the likelihood ratio of a beep is ln((1/2)/(1/4)) = ln(2) = 0.69. The posterior [...] are -2.94 + 0.69 = -2.23. This corresponds to an odds ratio of e^-2.23 = 0.108:1, and thus a probability of 0.108/1.108 = 0.097 = 9.7%. The correct answer is actually 9.5%. There was error due to rounding.
A: Log odds

Q: Logical rudeness
A: The primary type is probably the application of a theory of justified dismissal, such as a theory of error or insanity, to critics and dissenters. Another major type is the interpretation of criticism as behavior to be explained rather than answered. This is closely connected to the type that refuses to see a meta-level in the critic's criticism, and will not allow critics to escape the object-language of the theory... All the types have in common an evasion of a responsibility to answer criticism on the merits, when that evasion is authorized by the theory criticized. All types are triggered only by expounded criticism, and only insulate the proponent from conversion or capitulation, not the theory from refutation.
Suber goes on to say that rudeness does not imply falsehood, and in fact, to insulate yourself from criticism on the grounds that your opponent is rude is itself rude. He goes on to examine the competing norms of debate: we might wish to allow people freedom to hold to theories, including those which are rude, but we also wish debates to move forwards.

Q: The primary type is probably the application of a theory of justified dismissal, such as a theory of error or insanity, to critics and dissenters. Another major type is the interpretation of criticism as behavior to be explained rather than answered. This is closely connected to the type that refuses to see a meta-level in the critic's criticism, and will not allow critics to escape the object-language of the theory... All the types have in common an evasion of a responsibility to answer criticism on the merits, when that evasion is authorized by the theory criticized. All types are triggered only by expounded criticism, and only insulate the proponent from conversion or capitulation, not the theory from refutation.
Suber goes on to say that rudeness does not imply falsehood, and in fact, to insulate yourself from criticism on the grounds that your opponent is rude is itself rude. He goes on to examine the competing norms of debate: we might wish to allow people freedom to hold to theories, including those which are rude, but we also wish debates to move forwards.
A: Logical rudeness

Q: Loss aversion
A: Loss aversion is risk aversion's evil twin. A loss-averse agent tends to avoid uncertain gambles, not because every unit of money brings him a bit less utility, but because he weighs losses more heavily than gains, always treating his current level of money as somehow special.

Q: [...] is risk aversion's evil twin. A loss-averse agent tends to avoid uncertain gambles, not because every unit of money brings him a bit less utility, but because he weighs losses more heavily than gains, always treating his current level of money as somehow special.
A: Loss aversion

Q: Lottery
A: A tax on people who are bad at math. Also, a waste of hope. You will not win the lottery.

Q: A tax on people who are bad at math. Also, a waste of hope. You will not win the [...].
A: Lottery

Q: Luminosity
A: Luminosity is reflective awareness. A luminous mental state is one that you have and know that you have. It could be an emotion, a belief or alief, a disposition, a quale, a memory - anything that might happen or be stored in your brain. What's going on in your head?

Q: [...] is reflective awareness. A luminous mental state is one that you have and know that you have. It could be an emotion, a belief or alief, a disposition, a quale, a memory - anything that might happen or be stored in your brain. What's going on in your head?
A: Luminosity

Q: Magic
A: Marcello and I developed a convention in our AI work: when we ran into something we didn't understand, which was often, we would say "magic" - as in, "X magically does Y" - to remind ourselves that here was an unsolved problem, a gap in our understanding.
—Say Not "Complexity"
Traditional depictions of magic would seem to require introducing complex ontologically fundamental entities: some magician or sorceress says the right words and performs some ritual, and some part of the universe obeys their will. But how does it know when to obey someone's will? The stated conditions for the effect are far too complex to be implemented by a simple arrangement of mechanistic laws, the complexity of magic must be at least that of minds. What seems to humans like a simple explanation, sometimes isn't at all.
In our own naturalistic, reductionist universe, there is always a simpler explanation. Any complicated thing that happens, happens because there is some physical mechanism behind it, even if you don't know the mechanism yourself (which is most of the time). There is no magic.

Q: Marcello and I developed a convention in our AI work: when we ran into something we didn't understand, which was often, we would say "[...]" - as in, "X [...]ally does Y" - to remind ourselves that here was an unsolved problem, a gap in our understanding.
—Say Not "Complexity"
Traditional depictions of [...] would seem to require introducing complex ontologically fundamental entities: some [...]ian or sorceress says the right words and performs some ritual, and some part of the universe obeys their will. But how does it know when to obey someone's will? The stated conditions for the effect are far too complex to be implemented by a simple arrangement of mechanistic laws, the complexity of [...] must be at least that of minds. What seems to humans like a simple explanation, sometimes isn't at all.
In our own naturalistic, reductionist universe, there is always a simpler explanation. Any complicated thing that happens, happens because there is some physical mechanism behind it, even if you don't know the mechanism yourself (which is most of the time). There is no [...].
A: Magic

Q: Magical categories
A: A "magical category" is an English word which, although it sounds simple - hey, it's just one word, right? - is actually not simple, and furthermore, may be applied in a complicated way that drags in other considerations.
In Yudkowsky's work on Friendly AI, the notion of "magical categories" is particularly important with respect to English words that are supposed to describe morality and imperatives.  Suppose you say, for example, that "pleasure" is good.  You should instruct an AI to make people happy.  "All right," says the wise Friendly AI researcher, "so the AI inserts an electrode into your pleasure center to make you happy forever; or better yet, it disassembles you and uses your atoms to construct many tiny agents that are just complex enough to experience pleasure, and makes all of them happy."
"What?" you cry indignantly.  "That's not what I meant!  That's not true happiness!"
And if you try to unpack the concept of "true" happiness - that is, happiness which qualifies for being valuable - then you end up with a highly complicated Fun Theory.  So the word "happiness" actually turned out to have a complicated border; you would have to draw a squiggly surface in the space of possibilities, to capture everything you meant by "happiness" and exclude everything you didn't mean.
Or suppose that your mother's house was on fire, and so you wished to a literal-minded genie to "get my mother out of the house".  The genie said:  "What do you mean by that?  Can you specify it mathematically?"  And you replied:  "Increase the distance between the following contiguous physical entity, designated 'Mom', and the center of the following house."  So the genie takes your mother and throws her out of the Solar System.
If you were talking to a human firefighter, it wouldn't occur to him, any more than to you, to cause your mother to be outside the house by, say, exploding the gas main and sending her flaming form hurtling high into the air.  The firefighter understands explicitly that, in addition to wanting to increase the distance between your mother and the center of the burning house, you want your mother to survive... in good physical condition, and so on.  The firefighter, to some extent, shares your values; and so the invisible additional considerations that you're dragging into the problem, are also shared by the firefighter, and so can be omitted from your verbal instructions.
Physical brains are not powerful enough to search all possibilities; we have to cut down the search space to possibilities that are likely to be good.  Most of the "obviously bad" ways to get one's mother out of a burning building - those that would end up violating our other values, and so ranking very low in our preference ordering - do not even occur to us as possibilities.  When we tell the firefighter "My mother's in there!  Get her out!" it doesn't occur to us, or to the firefighter, to destroy the building with high explosives.  And so we don't realize that, in addition to the distance between our mother and the house, we're also dragging in highly complicated considerations having to do with the value of survival and good health, and what exactly constitutes being alive and healthy...

Q: A "magical category" is an English word which, although it sounds simple - hey, it's just one word, right? - is actually not simple, and furthermore, may be applied in a complicated way that drags in other considerations.
In Yudkowsky's work on Friendly AI, the notion of "[...]" is particularly important with respect to English words that are supposed to describe morality and imperatives.  Suppose you say, for example, that "pleasure" is good.  You should instruct an AI to make people happy.  "All right," says the wise Friendly AI researcher, "so the AI inserts an electrode into your pleasure center to make you happy forever; or better yet, it disassembles you and uses your atoms to construct many tiny agents that are just complex enough to experience pleasure, and makes all of them happy."
"What?" you cry indignantly.  "That's not what I meant!  That's not true happiness!"
And if you try to unpack the concept of "true" happiness - that is, happiness which qualifies for being valuable - then you end up with a highly complicated Fun Theory.  So the word "happiness" actually turned out to have a complicated border; you would have to draw a squiggly surface in the space of possibilities, to capture everything you meant by "happiness" and exclude everything you didn't mean.
Or suppose that your mother's house was on fire, and so you wished to a literal-minded genie to "get my mother out of the house".  The genie said:  "What do you mean by that?  Can you specify it mathematically?"  And you replied:  "Increase the distance between the following contiguous physical entity, designated 'Mom', and the center of the following house."  So the genie takes your mother and throws her out of the Solar System.
If you were talking to a human firefighter, it wouldn't occur to him, any more than to you, to cause your mother to be outside the house by, say, exploding the gas main and sending her flaming form hurtling high into the air.  The firefighter understands explicitly that, in addition to wanting to increase the distance between your mother and the center of the burning house, you want your mother to survive... in good physical condition, and so on.  The firefighter, to some extent, shares your values; and so the invisible additional considerations that you're dragging into the problem, are also shared by the firefighter, and so can be omitted from your verbal instructions.
Physical brains are not powerful enough to search all possibilities; we have to cut down the search space to possibilities that are likely to be good.  Most of the "obviously bad" ways to get one's mother out of a burning building - those that would end up violating our other values, and so ranking very low in our preference ordering - do not even occur to us as possibilities.  When we tell the firefighter "My mother's in there!  Get her out!" it doesn't occur to us, or to the firefighter, to destroy the building with high explosives.  And so we don't realize that, in addition to the distance between our mother and the house, we're also dragging in highly complicated considerations having to do with the value of survival and good health, and what exactly constitutes being alive and healthy...
A: Magical categories

Q: Making beliefs pay rent
A: Any belief should restrict which experiences to anticipate, to be potentially useful and thereby pay rent and earn its keep in your mind, so to speak. If a belief does not affect what you anticipate experiencing—if the world would look exactly the same whether the belief is true or whether it is false—then how could you possibly tell if it were false? And if there's no circumstance under which you would be able to notice your belief were false, then why do you believe it now?

Q: Any belief should restrict which experiences to anticipate, to be potentially useful and thereby pay rent and earn its keep in your mind, so to speak. If a belief does not affect what you anticipate experiencing—if the world would look exactly the same whether the belief is true or whether it is false—then how could you possibly tell if it were false? And if there's no circumstance under which you would be able to notice your belief were false, then why do you believe it now?
A: Making beliefs pay rent

Q: Many-worlds interpretation
A: Standard quantum mechanics is made of two parts: a part that describes the unitary and deterministic evolution of a state vector, and a part that describes how a state vector randomly collapses when subjected to "measurement".
The many-worlds interpretation (MWI) cuts away the latter part. It uses decoherence to explain how the universe splits into many separate branches, each of which looks like it came out of a random collapse.

Q: Standard quantum mechanics is made of two parts: a part that describes the unitary and deterministic evolution of a state vector, and a part that describes how a state vector randomly collapses when subjected to "measurement".
The [...] (MWI) cuts away the latter part. It uses decoherence to explain how the universe splits into many separate branches, each of which looks like it came out of a random collapse.
A: Many-worlds interpretation

Q: The map is not the territory
A: The map is not the territory metaphorically illustrates the differences between belief and reality. The phrase was coined by Alfred Korzybski. Our perception of the world is being generated by our brain and can be considered as a 'map' of reality written in neural paterns. Reality exists outside our mind but we can construct models of this 'territory' based on what we glimpse through our senses.
The methaphor is useful for illustrating several ideas in a more intuitive way:

Q: [...] metaphorically illustrates the differences between belief and reality. The phrase was coined by Alfred Korzybski. Our perception of the world is being generated by our brain and can be considered as a 'map' of reality written in neural paterns. Reality exists outside our mind but we can construct models of this 'territory' based on what we glimpse through our senses.
The methaphor is useful for illustrating several ideas in a more intuitive way:
A: The map is not the territory

Q: Map and territory
A: Less confusing than saying "belief and reality", "map and territory" reminds us that a map of Texas is not the same thing as Texas itself. Saying "map" also dispenses with possible meanings of "belief" apart from "representations of some part of reality".
Since our predictions don't always come true, we need different words to describe the thingy that generates our predictions and the thingy that generates our experimental results. The first thingy is called "belief", the second thingy "reality".

Q: Less confusing than saying "belief and reality", "[...]" reminds us that a map of Texas is not the same thing as Texas itself. Saying "map" also dispenses with possible meanings of "belief" apart from "representations of some part of reality".
Since our predictions don't always come true, we need different words to describe the thingy that generates our predictions and the thingy that generates our experimental results. The first thingy is called "belief", the second thingy "reality".
A: Map and territory

Q: Map and Territory (sequence)
A: A  collection of posts dealing with the fundamentals of rationality: the difference between the map and the territory, Bayes's Theorem and the nature of evidence, why anyone should care about truth, and minds as reflective cognitive engines.

Q: A  collection of posts dealing with the fundamentals of rationality: the difference between the map and the territory, Bayes's Theorem and the nature of evidence, why anyone should care about truth, and minds as reflective cognitive engines.
A: Map and Territory (sequence)

Q: Marginally zero-sum game
A: A zero-sum game where the efforts of each player not just give them a benefit at the expanse of the others, but decrease the efficacy of everyone's past and future actions, thus making everyone's actions extremely inefficient in the limit.

Q: A zero-sum game where the efforts of each player not just give them a benefit at the expanse of the others, but decrease the efficacy of everyone's past and future actions, thus making everyone's actions extremely inefficient in the limit.
A: Marginally zero-sum game

Q: Meme lineage
A: A meme lineage is a set of beliefs, attitudes, and practices that all share a clear common origin point. This concept also emphasizes the means of transmission of the beliefs in question. If a belief is part of a meme lineage that transmits for primarily social reasons, it may be discounted for purposes of the modesty argument.

Q: A [...] is a set of beliefs, attitudes, and practices that all share a clear common origin point. This concept also emphasizes the means of transmission of the beliefs in question. If a belief is part of a [...] that transmits for primarily social reasons, it may be discounted for purposes of the modesty argument.
A: Meme lineage

Q: Memorization
A: I suspect the most dangerous habit of thought taught in schools is that even if you don't really understand something, you should parrot it back anyway. One of the most fundamental life skills is realizing when you are confused, and school actively destroys this ability — teaches students that they "understand" when they can successfully answer questions on an exam, which is very very very far from absorbing the knowledge and making it a part of you. Students learn the habit that eating consists of putting food into mouth; the exams can't test for chewing or swallowing, and so they starve.
—Two More Things to Unlearn from School
Memorization is what you're doing when you cram for a university exam. It's not enough for understanding.

Q: I suspect the most dangerous habit of thought taught in schools is that even if you don't really understand something, you should parrot it back anyway. One of the most fundamental life skills is realizing when you are confused, and school actively destroys this ability — teaches students that they "understand" when they can successfully answer questions on an exam, which is very very very far from absorbing the knowledge and making it a part of you. Students learn the habit that eating consists of putting food into mouth; the exams can't test for chewing or swallowing, and so they starve.
—Two More Things to Unlearn from School
[...] is what you're doing when you cram for a university exam. It's not enough for understanding.
A: Memorization

Q: Mental models
A: A mental model is a hypothetical form of representation of knowledge in human mind. Mental models form to approximately describe dynamics of observed situations, and reuse parts of existing models to represent novel situations.

Q: A mental model is a hypothetical form of representation of knowledge in human mind. [...] form to approximately describe dynamics of observed situations, and reuse parts of existing models to represent novel situations.
A: Mental models

Q: Methods Of Rationality (fanfiction)
A: Harry Potter and the Methods of Rationality is a fanfiction by "Less Wrong" (a pen name of Eliezer Yudkowsky) with a premise that the orphaned Harry Potter has been raised by a scientist stepfather and now has the equivalent rationalist skill of roughly an 18-year-old Eliezer or thereabouts, i.e., massively flawed but still pretty darned impressive. Naturally the hero's challenges have been stepped up in difficulty as well.
It would be fair to understate that this fic has proven itself popular.
TV Tropes page here (warning, potential spoilers).
(This page also serves as a placeholder for subpages which can be linked to from fanfiction.net, so as to provide supplementary information.)

Q: Harry Potter and the Methods of Rationality is a fanfiction by "Less Wrong" (a pen name of Eliezer Yudkowsky) with a premise that the orphaned Harry Potter has been raised by a scientist stepfather and now has the equivalent rationalist skill of roughly an 18-year-old Eliezer or thereabouts, i.e., massively flawed but still pretty darned impressive. Naturally the hero's challenges have been stepped up in difficulty as well.
It would be fair to understate that this fic has proven itself popular.
TV Tropes page here (warning, potential spoilers).
(This page also serves as a placeholder for subpages which can be linked to from fanfiction.net, so as to provide supplementary information.)
A: Methods Of Rationality (fanfiction)

Q: Mind-killer
A: Mind-killer is a name given to topics (such as politics) that tend to produce extremely biased discussions.
It may be particularly important for Less Wrong to encourage a rational perspective with respect to these contentious topics.  Nevertheless, introducing these topics into an otherwise healthy discussion (for example, to present an analogy) may ruin it, encouraging fallacious modes of thinking.
  Politics as a mind killer
Politics is one obvious cause of mind-killer disputes:  Political disputes are not limited to standard disagreements about factual matters, nor to disputes of personality or perspective or even faction: they involve matters that people physically fight over in the real world—or at least, matters that are to be enforced by the government's monopoly of violence.  Accordingly, political discourse generally involves an adversarial process where careful deliberation is forgone; the focus shifts onto conflict management and on using arguments as soldiers to advance one's side.
We should expect that existing political allegiances of our users will add bias and irrationality to such discussions.  Politically-motivated people may also come here to Less Wrong in order to diligently reduce their casualties and add more recruits to their side.
For all of these reasons, Less Wrong tries to avoid particular political disputes.  Of course, discussing conflict reduction skills in the abstract is appropriate and encouraged!  In fact, it is the best way of preserving our deliberative, rationalist focus in the face of small-scale disputes and conflicts.
In addition, there may well be a balance of tensions between evaporative cooling of beliefs leading to groupthink, and extremely biased color politics. However, this is an underexplored issue.
  Social taboo
Another cause of mind-killers is social taboo. Negative connotations are associated with some topics, thus creating a strong bias supported by signaling drives that makes non-negative characterization of these topics appear absurd.

Q: [...] is a name given to topics (such as politics) that tend to produce extremely biased discussions.
It may be particularly important for Less Wrong to encourage a rational perspective with respect to these contentious topics.  Nevertheless, introducing these topics into an otherwise healthy discussion (for example, to present an analogy) may ruin it, encouraging fallacious modes of thinking.
  Politics as a mind killer
Politics is one obvious cause of [...] disputes:  Political disputes are not limited to standard disagreements about factual matters, nor to disputes of personality or perspective or even faction: they involve matters that people physically fight over in the real world—or at least, matters that are to be enforced by the government's monopoly of violence.  Accordingly, political discourse generally involves an adversarial process where careful deliberation is forgone; the focus shifts onto conflict management and on using arguments as soldiers to advance one's side.
We should expect that existing political allegiances of our users will add bias and irrationality to such discussions.  Politically-motivated people may also come here to Less Wrong in order to diligently reduce their casualties and add more recruits to their side.
For all of these reasons, Less Wrong tries to avoid particular political disputes.  Of course, discussing conflict reduction skills in the abstract is appropriate and encouraged!  In fact, it is the best way of preserving our deliberative, rationalist focus in the face of small-scale disputes and conflicts.
In addition, there may well be a balance of tensions between evaporative cooling of beliefs leading to groupthink, and extremely biased color politics. However, this is an underexplored issue.
  Social taboo
Another cause of [...]s is social taboo. Negative connotations are associated with some topics, thus creating a strong bias supported by signaling drives that makes non-negative characterization of these topics appear absurd.
A: Mind-killer

Q: Mind design space
A: Mind design space refers to the configuration space of possible minds. As humans living in a human world, we can safely make all sorts of assumptions about the minds around us without even realizing it. Each human might have their own unique personal qualities, so it might naively seem that there's nothing you can say about people you don't know. But there's actually quite a lot you can say (with high or very high probability) about a random human: that they have standard emotions like happiness, sadness, and anger; standard senses like sight, vision, and hearing; that they speak a language; and no doubt any number of other subtle features that are even harder to quickly explain in words. These things are the specific results of adaptation pressures in the ancestral environment and can't be expected to be shared by a random alien or AI. That is, humans are packed into a tiny dot in the configuration space: there is vast range over of other ways a mind can be.

Q: [...] refers to the configuration space of possible minds. As humans living in a human world, we can safely make all sorts of assumptions about the minds around us without even realizing it. Each human might have their own unique personal qualities, so it might naively seem that there's nothing you can say about people you don't know. But there's actually quite a lot you can say (with high or very high probability) about a random human: that they have standard emotions like happiness, sadness, and anger; standard senses like sight, vision, and hearing; that they speak a language; and no doubt any number of other subtle features that are even harder to quickly explain in words. These things are the specific results of adaptation pressures in the ancestral environment and can't be expected to be shared by a random alien or AI. That is, humans are packed into a tiny dot in the configuration space: there is vast range over of other ways a mind can be.
A: Mind design space

Q: Mind projection fallacy
A: Confusion exists in our minds, not in reality. A blank spot on your map does not correspond to a blank territory.
Our minds are not transparent windows unto veridical reality; when you look at a rock, you experience not the the rock itself, but your mind's representation of the rock, reconstructed from photons bouncing off its surface. Sugar in and of itself is not inherently sweet; the sugar itself only has the chemical properties that it does, which your brain interprets as sweet.
Physicist and Bayesian philosopher E.T. Jaynes coined the term mind projection fallacy to refer to this kind of failure to distinguish between epistemological claims (statements about belief, about your map, about what we can say about reality) and ontological claims (statements about reality, about the territory, about how things are). In particular, the concept was applied in the critique of frequentist interpretation of the notion of probability as a property of physical systems rather than an epistemic device concerned with levels of certainty, Bayesian probability.

Q: Confusion exists in our minds, not in reality. A blank spot on your map does not correspond to a blank territory.
Our minds are not transparent windows unto veridical reality; when you look at a rock, you experience not the the rock itself, but your mind's representation of the rock, reconstructed from photons bouncing off its surface. Sugar in and of itself is not inherently sweet; the sugar itself only has the chemical properties that it does, which your brain interprets as sweet.
Physicist and Bayesian philosopher E.T. Jaynes coined the term [...] to refer to this kind of failure to distinguish between epistemological claims (statements about belief, about your map, about what we can say about reality) and ontological claims (statements about reality, about the territory, about how things are). In particular, the concept was applied in the critique of frequentist interpretation of the notion of probability as a property of physical systems rather than an epistemic device concerned with levels of certainty, Bayesian probability.
A: Mind projection fallacy

Q: Modesty
A: Admitting or boasting of flaws so as to not create perceptions of arrogance. Not to be confused with humility.

Q: Admitting or boasting of flaws so as to not create perceptions of arrogance. Not to be confused with humility.
A: Modesty

Q: Modesty argument
A: The modesty argument is the claim that when two or more rational agents have common knowledge of a disagreement over the likelihood of an issue of simple fact, they should each adjust their probability estimates in the direction of the others'. This process should continue until the two agents are in full agreement. The name comes from the idea that rational agents should not privilege evidence they have gathered over evidence gathered by others, i.e. agents should be modest about their own estimates.
The Modesty Argument is inspired by Aumann's agreement theorem, which shows that genuine Bayesians cannot agree to disagree.

Q: The [...] is the claim that when two or more rational agents have common knowledge of a disagreement over the likelihood of an issue of simple fact, they should each adjust their probability estimates in the direction of the others'. This process should continue until the two agents are in full agreement. The name comes from the idea that rational agents should not privilege evidence they have gathered over evidence gathered by others, i.e. agents should be modest about their own estimates.
The Modesty Argument is inspired by Aumann's agreement theorem, which shows that genuine Bayesians cannot agree to disagree.
A: Modesty argument

Q: Motivated cognition
A: An academic/technical term for various mental processes that lead to desired conclusions regardless of the veracity of those conclusions.

Q: An academic/technical term for various mental processes that lead to desired conclusions regardless of the veracity of those conclusions.
A: Motivated cognition

Q: Motivated skepticism
A: Motivated skepticism is the mistake of applying more skepticism to claims that you don't like (or intuitively disbelieve), than to claims that you do like. Because emotional disposition towards a claim isn't generally evidence about its truth, including it in the process of arriving at a belief means holding the belief partly for reasons other than because it's true.

Q: [...] is the mistake of applying more skepticism to claims that you don't like (or intuitively disbelieve), than to claims that you do like. Because emotional disposition towards a claim isn't generally evidence about its truth, including it in the process of arriving at a belief means holding the belief partly for reasons other than because it's true.
A: Motivated skepticism

Q: Mysterious Answers to Mysterious Questions
A: A sequence on how to see through the disguises of answers or beliefs or statements, that don't answer or say or mean anything.
Mysterious Answers to Mysterious Questions is probably the most important core sequence in Less Wrong.  Posts in the sequence are distributed from 28 Jul 07 to 11 Sep 07.
Contents
1 Main sequence
1.1 Making Beliefs Pay Rent (in Anticipated Experiences)
1.2 Belief in Belief
1.3 Bayesian Judo
1.4 Professing and Cheering
1.5 Belief as Attire
1.6 Focus Your Uncertainty
1.7 The Virtue of Narrowness
1.8 Your Strength As A Rationalist
1.9 Absence of Evidence is Evidence of Absence
1.10 Conservation of Expected Evidence
1.11 Hindsight Bias
1.12 Hindsight Devalues Science
1.13 Fake Explanations
1.14 Guessing the Teacher's Password
1.15 Science as Attire
1.16 Fake Causality
1.17 Semantic Stopsigns
1.18 Mysterious Answers to Mysterious Questions
1.19 The Futility of Emergence
1.20 Say Not "Complexity"
1.21 Positive Bias: Look Into the Dark
1.22 My Wild and Reckless Youth
1.23 Failing to Learn from History
1.24 Making History Available
1.25 Explain/Worship/Ignore?
1.26 "Science" as Curiosity-Stopper
1.27 Applause Lights
1.28 Truly Part of You
1.29 Chaotic Inversion
2 See also
3 Alternative formats
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Main sequence
Making Beliefs Pay Rent (in Anticipated Experiences)
Not every belief that we have is directly about sensory experience, but beliefs should pay rent in anticipations of experience.  For example, if I believe that "Gravity is 9.8 m/s^2" then I should be able to predict where I'll see the second hand on my watch at the time I hear the crash of a bowling ball dropped off a building.  On the other hand, if your postmodern English professor says that the famous writer Wulky is a "post-utopian", this may not actually mean anything.  The moral is to ask "What experiences do I anticipate?" not "What statements do I believe?"
Belief in Belief
Suppose someone claims to have a dragon in their garage, but as soon as you go to look, they say, "It's an invisible dragon!"  The remarkable thing is that they know in advance exactly which experimental results they shall have to excuse, indicating that some part of their mind knows what's really going on.  And yet they may honestly believe they believe there's a dragon in the garage.  They may perhaps believe it is virtuous to believe there is a dragon in the garage, and believe themselves virtuous.  Even though they anticipate as if there is no dragon.
Bayesian Judo
You can have some fun with people whose anticipations get out of sync with what they believe they believe. This post recounts a conversation in which a theist had to backpedal when he realized that, by drawing an empirical inference from his religion, he had opened up his religion to empirical disproof.
Professing and Cheering
A woman on a panel enthusiastically declared her belief in a pagan creation myth, flaunting its most outrageously improbable elements.  This seemed weirder than "belief in belief" (she didn't act like she needed validation) or "religious profession" (she didn't try to act like she took her religion seriously).  So, what was she doing? She was cheering for paganism — cheering loudly by making ridiculous claims.
Belief as Attire
When you've stopped anticipating-as-if something, but still believe it is virtuous to believe it, this does not create the true fire of the child who really does believe.  On the other hand, it is very easy for people to be passionate about group identification - sports teams, political sports teams - and this may account for the passion of beliefs worn as team-identification attire.
Focus Your Uncertainty
Justifies the use of subjective probability estimates. Let's say you get paid to explain movements of the financial markets after the fact. You'd like to prepare your explanations for each way things could go in advance, and you can do your job better if you spend more time on preparing explanations for outcomes that are actually more likely. Being able to estimate probabilities could be useful even if you get paid to explain anything.
The Virtue of Narrowness
It was perfectly all right for Isaac Newton to explain just gravity, just the way things fall down - and how planets orbit the Sun, and how the Moon generates the tides - but not the role of money in human society or how the heart pumps blood.  Sneering at narrowness is rather reminiscent of ancient Greeks who thought that going out and actually looking at things was manual labor, and manual labor was for slaves.
Your Strength As A Rationalist
A hypothesis that forbids nothing, permits everything, and thereby fails to constrain anticipation.  Your strength as a rationalist is your ability to be more confused by fiction than by reality.  If you are equally good at explaining any outcome, you have zero knowledge.
Absence of Evidence is Evidence of Absence
Absence of proof is not proof of absence.  But absence of evidence is always evidence of absence.  According to the probability calculus, if P(H|E) > P(H) (observing E would be evidence for hypothesis H), then P(H|~E) < P(H) (absence of E is evidence against H).  The absence of evidence may be a strong indicator or a weak indicator that the hypothesis is false, but it's always an indicator.
Conservation of Expected Evidence
If you are about to make an observation, then the expected value of your posterior probability must equal your current prior probability.  On average, you must expect to be exactly as confident as when you started out.  If you are a true Bayesian, you cannot seek evidence to confirm your theory, because you do not expect any evidence to do that.  You can only seek evidence to test your theory.
Hindsight Bias
Hindsight bias makes us overestimate how well our model could have predicted a known outcome.   We underestimate the cost of avoiding a known bad outcome, because we forget that many other equally severe outcomes seemed as probable at the time.  Hindsight bias distorts the testing of our models by observation, making us think that our models are better than they really are.
Hindsight Devalues Science
Hindsight bias leads us to systematically undervalue scientific findings, because we find it too easy to retrofit them into our models of the world. This unfairly devalues the contributions of researchers.  Worse, it prevents us from noticing when we are seeing evidence that doesn't fit what we really would have expected.  We need to make a conscious effort to be shocked enough.
Fake Explanations
People think that fake explanations use words like "magic", while real explanations use scientific words like "heat conduction". But being a real explanation isn't a matter of literary genre. Scientific-sounding words aren't enough. Real explanations constrain anticipation. Ideally, you could explain only the observations that actually happened. Fake explanations could just as well "explain" the opposite of what you observed.
Guessing the Teacher's Password
In schools, "education" often consists of having students memorize answers to specific questions (i.e., the "teacher's password"), rather than learning a predictive model that says what is and isn't likely to happen. Thus, students incorrectly learn to guess at passwords in the face of strange observations rather than admit their confusion. Don't do that: any explanation you give should have a predictive model behind it. If your explanation lacks such a model, start from a recognition of your own confusion and surprise at seeing the result.
Science as Attire
Fake Causality
Semantic Stopsigns
When seemingly unanswerable questions are answered with, say, "God did it," that answer doesn't resolve the question, but it does tell you to stop asking further questions; it functions as a semantic stopsign. But religion is by no means the only source of semantic stopsigns. If you're tempted to solve any problem or explain any event with a word like "government" or "big business" or "terrorism," and you fail to ask the obvious next question "How exactly does [government|business|terrorism] explain this thing or solve this problem?", then that word is a semantic stopsign for you.
Mysterious Answers to Mysterious Questions
Sometimes it seems that people want their semantic stopsigns not to actually explain anything, because they like feeling that the universe is too grand and mysterious to understand. But nothing is mysterious in itself; if we can't understand the universe, it's not because the universe is grand but because we are ignorant, and there's nothing wonderful about that ignorance.
The Futility of Emergence
Say Not "Complexity"
Positive Bias: Look Into the Dark
The 2-4-6 experiment suggests that humans tend to look for positive evidence ("My theory predicts this, and it happens!") rather than negative evidence ("My theory predicts this won't happen, and it doesn't.") This is similar to, but separate from, confirmation bias. To spot an explanation that isn't helpful, it's not enough to think of what it does explain very well — you also have to search for results it couldn't explain.
My Wild and Reckless Youth
Failing to Learn from History
Making History Available
Explain/Worship/Ignore?
"Science" as Curiosity-Stopper
Although science does have explanations for phenomena, it is not enough to simply say that "Science!" is responsible for how something works -- nor is it enough to appeal to something more specific like "electricity" or "conduction".  Yet for many people, simply noting that "Science has an answer" is enough to make them no longer curious about how it works.  In that respect, "Science" is no different from more blatant curiosity-stoppers like "God did it!"  But you shouldn't let your interest die simply because someone else knows the answer (which is a rather strange heuristic anyway): You should only be satisfied with a predictive model, and how a given phenomenon fits into that model.
Applause Lights
Policy proposals need to come with specifics, not just virtuous-sounding words like "democracy" or "balance". These words can stand for specific proposals (democracy: resolving conflicts through voting) but they are often used in an uninformative way to convey mysterious goodness. To test whether a proposal actually carries new information, try reversing it.
Truly Part of You
How much of your knowledge could you regenerate, if it were deleted from your mind?  If you don't have enough experience to regenerate beliefs when they are deleted, then do you have enough experience to connect that belief to anything at all?

Q: A sequence on how to see through the disguises of answers or beliefs or statements, that don't answer or say or mean anything.
[...] is probably the most important core sequence in Less Wrong.  Posts in the sequence are distributed from 28 Jul 07 to 11 Sep 07.
Contents
1 Main sequence
1.1 Making Beliefs Pay Rent (in Anticipated Experiences)
1.2 Belief in Belief
1.3 Bayesian Judo
1.4 Professing and Cheering
1.5 Belief as Attire
1.6 Focus Your Uncertainty
1.7 The Virtue of Narrowness
1.8 Your Strength As A Rationalist
1.9 Absence of Evidence is Evidence of Absence
1.10 Conservation of Expected Evidence
1.11 Hindsight Bias
1.12 Hindsight Devalues Science
1.13 Fake Explanations
1.14 Guessing the Teacher's Password
1.15 Science as Attire
1.16 Fake Causality
1.17 Semantic Stopsigns
1.18 [...]
1.19 The Futility of Emergence
1.20 Say Not "Complexity"
1.21 Positive Bias: Look Into the Dark
1.22 My Wild and Reckless Youth
1.23 Failing to Learn from History
1.24 Making History Available
1.25 Explain/Worship/Ignore?
1.26 "Science" as Curiosity-Stopper
1.27 Applause Lights
1.28 Truly Part of You
1.29 Chaotic Inversion
2 See also
3 Alternative formats
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Main sequence
Making Beliefs Pay Rent (in Anticipated Experiences)
Not every belief that we have is directly about sensory experience, but beliefs should pay rent in anticipations of experience.  For example, if I believe that "Gravity is 9.8 m/s^2" then I should be able to predict where I'll see the second hand on my watch at the time I hear the crash of a bowling ball dropped off a building.  On the other hand, if your postmodern English professor says that the famous writer Wulky is a "post-utopian", this may not actually mean anything.  The moral is to ask "What experiences do I anticipate?" not "What statements do I believe?"
Belief in Belief
Suppose someone claims to have a dragon in their garage, but as soon as you go to look, they say, "It's an invisible dragon!"  The remarkable thing is that they know in advance exactly which experimental results they shall have to excuse, indicating that some part of their mind knows what's really going on.  And yet they may honestly believe they believe there's a dragon in the garage.  They may perhaps believe it is virtuous to believe there is a dragon in the garage, and believe themselves virtuous.  Even though they anticipate as if there is no dragon.
Bayesian Judo
You can have some fun with people whose anticipations get out of sync with what they believe they believe. This post recounts a conversation in which a theist had to backpedal when he realized that, by drawing an empirical inference from his religion, he had opened up his religion to empirical disproof.
Professing and Cheering
A woman on a panel enthusiastically declared her belief in a pagan creation myth, flaunting its most outrageously improbable elements.  This seemed weirder than "belief in belief" (she didn't act like she needed validation) or "religious profession" (she didn't try to act like she took her religion seriously).  So, what was she doing? She was cheering for paganism — cheering loudly by making ridiculous claims.
Belief as Attire
When you've stopped anticipating-as-if something, but still believe it is virtuous to believe it, this does not create the true fire of the child who really does believe.  On the other hand, it is very easy for people to be passionate about group identification - sports teams, political sports teams - and this may account for the passion of beliefs worn as team-identification attire.
Focus Your Uncertainty
Justifies the use of subjective probability estimates. Let's say you get paid to explain movements of the financial markets after the fact. You'd like to prepare your explanations for each way things could go in advance, and you can do your job better if you spend more time on preparing explanations for outcomes that are actually more likely. Being able to estimate probabilities could be useful even if you get paid to explain anything.
The Virtue of Narrowness
It was perfectly all right for Isaac Newton to explain just gravity, just the way things fall down - and how planets orbit the Sun, and how the Moon generates the tides - but not the role of money in human society or how the heart pumps blood.  Sneering at narrowness is rather reminiscent of ancient Greeks who thought that going out and actually looking at things was manual labor, and manual labor was for slaves.
Your Strength As A Rationalist
A hypothesis that forbids nothing, permits everything, and thereby fails to constrain anticipation.  Your strength as a rationalist is your ability to be more confused by fiction than by reality.  If you are equally good at explaining any outcome, you have zero knowledge.
Absence of Evidence is Evidence of Absence
Absence of proof is not proof of absence.  But absence of evidence is always evidence of absence.  According to the probability calculus, if P(H|E) > P(H) (observing E would be evidence for hypothesis H), then P(H|~E) < P(H) (absence of E is evidence against H).  The absence of evidence may be a strong indicator or a weak indicator that the hypothesis is false, but it's always an indicator.
Conservation of Expected Evidence
If you are about to make an observation, then the expected value of your posterior probability must equal your current prior probability.  On average, you must expect to be exactly as confident as when you started out.  If you are a true Bayesian, you cannot seek evidence to confirm your theory, because you do not expect any evidence to do that.  You can only seek evidence to test your theory.
Hindsight Bias
Hindsight bias makes us overestimate how well our model could have predicted a known outcome.   We underestimate the cost of avoiding a known bad outcome, because we forget that many other equally severe outcomes seemed as probable at the time.  Hindsight bias distorts the testing of our models by observation, making us think that our models are better than they really are.
Hindsight Devalues Science
Hindsight bias leads us to systematically undervalue scientific findings, because we find it too easy to retrofit them into our models of the world. This unfairly devalues the contributions of researchers.  Worse, it prevents us from noticing when we are seeing evidence that doesn't fit what we really would have expected.  We need to make a conscious effort to be shocked enough.
Fake Explanations
People think that fake explanations use words like "magic", while real explanations use scientific words like "heat conduction". But being a real explanation isn't a matter of literary genre. Scientific-sounding words aren't enough. Real explanations constrain anticipation. Ideally, you could explain only the observations that actually happened. Fake explanations could just as well "explain" the opposite of what you observed.
Guessing the Teacher's Password
In schools, "education" often consists of having students memorize answers to specific questions (i.e., the "teacher's password"), rather than learning a predictive model that says what is and isn't likely to happen. Thus, students incorrectly learn to guess at passwords in the face of strange observations rather than admit their confusion. Don't do that: any explanation you give should have a predictive model behind it. If your explanation lacks such a model, start from a recognition of your own confusion and surprise at seeing the result.
Science as Attire
Fake Causality
Semantic Stopsigns
When seemingly unanswerable questions are answered with, say, "God did it," that answer doesn't resolve the question, but it does tell you to stop asking further questions; it functions as a semantic stopsign. But religion is by no means the only source of semantic stopsigns. If you're tempted to solve any problem or explain any event with a word like "government" or "big business" or "terrorism," and you fail to ask the obvious next question "How exactly does [government|business|terrorism] explain this thing or solve this problem?", then that word is a semantic stopsign for you.
[...]
Sometimes it seems that people want their semantic stopsigns not to actually explain anything, because they like feeling that the universe is too grand and mysterious to understand. But nothing is mysterious in itself; if we can't understand the universe, it's not because the universe is grand but because we are ignorant, and there's nothing wonderful about that ignorance.
The Futility of Emergence
Say Not "Complexity"
Positive Bias: Look Into the Dark
The 2-4-6 experiment suggests that humans tend to look for positive evidence ("My theory predicts this, and it happens!") rather than negative evidence ("My theory predicts this won't happen, and it doesn't.") This is similar to, but separate from, confirmation bias. To spot an explanation that isn't helpful, it's not enough to think of what it does explain very well — you also have to search for results it couldn't explain.
My Wild and Reckless Youth
Failing to Learn from History
Making History Available
Explain/Worship/Ignore?
"Science" as Curiosity-Stopper
Although science does have explanations for phenomena, it is not enough to simply say that "Science!" is responsible for how something works -- nor is it enough to appeal to something more specific like "electricity" or "conduction".  Yet for many people, simply noting that "Science has an answer" is enough to make them no longer curious about how it works.  In that respect, "Science" is no different from more blatant curiosity-stoppers like "God did it!"  But you shouldn't let your interest die simply because someone else knows the answer (which is a rather strange heuristic anyway): You should only be satisfied with a predictive model, and how a given phenomenon fits into that model.
Applause Lights
Policy proposals need to come with specifics, not just virtuous-sounding words like "democracy" or "balance". These words can stand for specific proposals (democracy: resolving conflicts through voting) but they are often used in an uninformative way to convey mysterious goodness. To test whether a proposal actually carries new information, try reversing it.
Truly Part of You
How much of your knowledge could you regenerate, if it were deleted from your mind?  If you don't have enough experience to regenerate beliefs when they are deleted, then do you have enough experience to connect that belief to anything at all?
A: Mysterious Answers to Mysterious Questions

Q: Nanotechnology
A: Nanotechnology is a hypothetical set of future developments, centered around devices with parts that are on the scale of less than 100 nanometers.  Nanotechnology was originally  proposed by Richard Feynman[1] and formalized by Eric Drexler[2].

Q: [...] is a hypothetical set of future developments, centered around devices with parts that are on the scale of less than 100 nanometers.  [...] was originally  proposed by Richard Feynman[1] and formalized by Eric Drexler[2].
A: Nanotechnology

Q: Narrative fallacy
A: The narrative fallacy addresses our limited ability to look at sequences of facts without weaving an explanation into them, or, equivalently, forcing a logical link, an arrow of relationship upon them. Explanations bind facts together. They make them all the more easily remembered; they help them make more sense. Where this propensity can go wrong is when it increases our impression of understanding.

Q: The [...] addresses our limited ability to look at sequences of facts without weaving an explanation into them, or, equivalently, forcing a logical link, an arrow of relationship upon them. Explanations bind facts together. They make them all the more easily remembered; they help them make more sense. Where this propensity can go wrong is when it increases our impression of understanding.
A: Narrative fallacy

Q: Near and far thinking
A: Near and far are two modes (or a spectrum of modes) in which we can think about things. We choose which mode to think about something in based on its distance from us, or on the level of detail we need. This property of human mind is studied in construal level theory.

Q: Near and far are two modes (or a spectrum of modes) in which we can think about things. We choose which mode to think about something in based on its distance from us, or on the level of detail we need. This property of human mind is studied in construal level theory.
A: Near and far thinking

Q: Newcomb\'s problem
A: In Newcomb's problem, a superintelligence called Omega shows you two boxes, A and B, and offers you the choice of taking only box A, or both boxes A and B.   Omega has put  $1,000 in box  B.   If  Omega thinks you will take box A only, he has put $1,000,000 in it.  Otherwise he  has left it empty.  Omega has played this game many times, and has never been wrong in his predictions about whether someone will take both boxes or not.
Terms used in relation to this paradox:
Omega, the superintelligence who decides whether to put the million in box A.
one-box: to take only box A
two-box: to take both boxes
A succinct introduction to analysis of the paradox, paraphrased from Gary Drescher's Good and Real:
What makes this a "paradox" is that it brings into sharp conflict two distinct intuitions we have about decision-making, which rarely bear on the same situation but clash in the case of Newcomb's. The first intution is considering rational expectations, act so as to bring about desired outcomes. This suggests one-boxing: we expect, based on the evidence, that we will find box A empty if we two-box. The second intuition is only act if your action will alter the outcome. This suggests two-boxing: our decision to take one box or both cannot alter the outcome, which robs the first intuition of its power to suggest one-boxing; since two-boxing has strictly greater expected utility, we choose that.
Irrelevance of Omega's physical impossibility
Sometimes people dismiss Newcomb's problem because a being such as Omega is physically impossible. Actually, the possibility or impossibility of Omega is irrelevant. Consider a skilled human psychologist that can predict other humans' actions with, say, 65% accuracy. Now imagine they start running Newcomb trials with themselves as Omega.

Q: In [...], a superintelligence called Omega shows you two boxes, A and B, and offers you the choice of taking only box A, or both boxes A and B.   Omega has put  $1,000 in box  B.   If  Omega thinks you will take box A only, he has put $1,000,000 in it.  Otherwise he  has left it empty.  Omega has played this game many times, and has never been wrong in his predictions about whether someone will take both boxes or not.
Terms used in relation to this paradox:
Omega, the superintelligence who decides whether to put the million in box A.
one-box: to take only box A
two-box: to take both boxes
A succinct introduction to analysis of the paradox, paraphrased from Gary Drescher's Good and Real:
What makes this a "paradox" is that it brings into sharp conflict two distinct intuitions we have about decision-making, which rarely bear on the same situation but clash in the case of Newcomb's. The first intution is considering rational expectations, act so as to bring about desired outcomes. This suggests one-boxing: we expect, based on the evidence, that we will find box A empty if we two-box. The second intuition is only act if your action will alter the outcome. This suggests two-boxing: our decision to take one box or both cannot alter the outcome, which robs the first intuition of its power to suggest one-boxing; since two-boxing has strictly greater expected utility, we choose that.
Irrelevance of Omega's physical impossibility
Sometimes people dismiss [...] because a being such as Omega is physically impossible. Actually, the possibility or impossibility of Omega is irrelevant. Consider a skilled human psychologist that can predict other humans' actions with, say, 65% accuracy. Now imagine they start running Newcomb trials with themselves as Omega.
A: Newcomb\'s problem

Q: No-Nonsense Metaethics
A: A sequence by lukeprog that explains and defends a naturalistic approach to metaethics.

Q: A sequence by lukeprog that explains and defends a naturalistic approach to metaethics.
A: No-Nonsense Metaethics

Q: No one knows what science doesn\'t know
A: In a technological civilization containing billions of people (a number you are not equipped to understand intuitively), there is not and cannot be any central authority that oversees all human knowledge. On many questions, no one can say that science doesn't know something, because far too much scientific information has been amassed for any one person or small team to sort through it all. Just because you haven't read about it, doesn't mean no one in the depths of human history has investigated this same question and come away with genuine answers.

Q: In a technological civilization containing billions of people (a number you are not equipped to understand intuitively), there is not and cannot be any central authority that oversees all human knowledge. On many questions, no one can say that science doesn't know something, because far too much scientific information has been amassed for any one person or small team to sort through it all. Just because you haven't read about it, doesn't mean no one in the depths of human history has investigated this same question and come away with genuine answers.
A: No one knows what science doesn\'t know

Q: No safe defense
A: In this crazy world, there's no simple procedure you can follow, no finite checklist or fixed ritual you can obey to ensure that your reasoning has been defensible and your actions just. There is a sense in which you are never safe. There's no one you can trust; the full force of your skepticism must be applied to everything. It may be tempting to seek out some authoritative book, like the Bible, or the Feynman Lectures on Physics, and trust that everything in the book is true. But the book is only probabilistic evidence about the state of the world, however strong. Authorities can be trusted exactly as much as a rational evaluation of the evidence deems them trustworthy, no more and no less.

Q: In this crazy world, there's no simple procedure you can follow, no finite checklist or fixed ritual you can obey to ensure that your reasoning has been defensible and your actions just. There is a sense in which you are never safe. There's no one you can trust; the full force of your skepticism must be applied to everything. It may be tempting to seek out some authoritative book, like the Bible, or the Feynman Lectures on Physics, and trust that everything in the book is true. But the book is only probabilistic evidence about the state of the world, however strong. Authorities can be trusted exactly as much as a rational evaluation of the evidence deems them trustworthy, no more and no less.
A: No safe defense

Q: Nonapples
A: Any object, tool, technique, or theory which is defined only as being "not this specific one".
The set "apples" is much narrower than the set "not apples".  Apples form a compact cluster in thingspace, but nonapples vary much more widely in price, and size, and use.  When you say to build a wagon using "wood", you're giving much more concrete advice than when you say "not wood".  There are different kinds of wood, of course - but even so, when you say "wood", you've narrowed down the range of possible building materials a whole lot more than when you say "not wood".

Q: Any object, tool, technique, or theory which is defined only as being "not this specific one".
The set "apples" is much narrower than the set "not apples".  Apples form a compact cluster in thingspace, but [...] vary much more widely in price, and size, and use.  When you say to build a wagon using "wood", you're giving much more concrete advice than when you say "not wood".  There are different kinds of wood, of course - but even so, when you say "wood", you've narrowed down the range of possible building materials a whole lot more than when you say "not wood".
A: Nonapples

Q: Not technically a lie
A: Promoting less than maximally accurate beliefs is an act of sabotage. Don't do it to anyone unless you'd also slash their tires.
—Black Belt Bayesian
A statement that is literally true, but causes the listener to attain false beliefs by performing incorrect inference, is not technically a lie. Since the purpose of communication is to transfer beliefs to the other person's mind, there is little practical distinction between lying and not technically lying. Depending on circumstances, one may be preferable to another, but only by a slim margin. A weapon of dark arts, or an attire for someone identifying with truth-telling.

Q: Promoting less than maximally accurate beliefs is an act of sabotage. Don't do it to anyone unless you'd also slash their tires.
—Black Belt Bayesian
A statement that is literally true, but causes the listener to attain false beliefs by performing incorrect inference, is [...]. Since the purpose of communication is to transfer beliefs to the other person's mind, there is little practical distinction between lying and not technically lying. Depending on circumstances, one may be preferable to another, but only by a slim margin. A weapon of dark arts, or an attire for someone identifying with truth-telling.
A: Not technically a lie

Q: Occam\'s razor
A: Occam's razor is a principle commonly stated as "Entities must not be multiplied beyond necessity". When several theories are able to explain the same observations, Occam's razor suggests the simpler one is preferable. It must be noted that Occam's razor is a requirement for the simplicity of theories, not for the size of the systems described by those theories. For example, the immensity of the Universe isn't at odds with the principle of Occam's razor.
Occam's razor is necessitated by the conjunction rule of probability theory: the conjunction A and B is necessarily less (or equally, in the case of logical equivalence) probable than the A alone; every detail you tack onto your story drives the probability down.
Occam's razor has been formalized as Minimum Description Length or Minimum Message Length, in which the total size of the theory is the length of the message required to describe the theory, plus the length of the message required to describe the evidence using the theory.  Solomonoff induction is the ultimate case of minimum message length in which the code for messages can describe all computable hypotheses.  This has jokingly been referred to as "Solomonoff's lightsaber".

Q: [...] is a principle commonly stated as "Entities must not be multiplied beyond necessity". When several theories are able to explain the same observations, [...] suggests the simpler one is preferable. It must be noted that [...] is a requirement for the simplicity of theories, not for the size of the systems described by those theories. For example, the immensity of the Universe isn't at odds with the principle of [...].
[...] is necessitated by the conjunction rule of probability theory: the conjunction A and B is necessarily less (or equally, in the case of logical equivalence) probable than the A alone; every detail you tack onto your story drives the probability down.
[...] has been formalized as Minimum Description Length or Minimum Message Length, in which the total size of the theory is the length of the message required to describe the theory, plus the length of the message required to describe the evidence using the theory.  Solomonoff induction is the ultimate case of minimum message length in which the code for messages can describe all computable hypotheses.  This has jokingly been referred to as "Solomonoff's lightsaber".
A: Occam\'s razor

Q: Odds ratio
A: Odds ratios are an alternate way of expressing probabilities, which simplifies the process of updating them with new evidence. The odds ratio of A is P(A)/P(¬A).
Thus, in order to find the posterior odds ratio , one simply multiplies the prior odds ratio  by the likelihood ratio .
Odds ratios are commonly written as the ratio of two numbers separated by a colon. For example, if P(A) = 2/3, the odds ratio would be 2, but this would most likely be written as 2:1.
The relation between odds ratio, a:b, and probability, p is as follows:
a:b = p:(1 − p)
Suppose you have a box that has a 5% chance of containing a diamond. You also have a diamond detector that beeps half of the time if there is a diamond, and one fourth of the time if there is not. You wave the diamond detector over the box and it beeps.
The prior odds of the box containing a diamond are 1:19. The likelihood ratio of a beep is 1/2:1/4 = 2:1. The posterior odds are 1:19 * 2:1 = 2:19. This corresponds to about a probability of 2/21, which is about 0.095 or 9.5%.

Q: [...]s are an alternate way of expressing probabilities, which simplifies the process of updating them with new evidence. The [...] of A is P(A)/P(¬A).
Thus, in order to find the posterior [...] , one simply multiplies the prior [...]  by the likelihood ratio .
[...]s are commonly written as the ratio of two numbers separated by a colon. For example, if P(A) = 2/3, the [...] would be 2, but this would most likely be written as 2:1.
The relation between [...], a:b, and probability, p is as follows:
a:b = p:(1 − p)
Suppose you have a box that has a 5% chance of containing a diamond. You also have a diamond detector that beeps half of the time if there is a diamond, and one fourth of the time if there is not. You wave the diamond detector over the box and it beeps.
The prior odds of the box containing a diamond are 1:19. The likelihood ratio of a beep is 1/2:1/4 = 2:1. The posterior odds are 1:19 * 2:1 = 2:19. This corresponds to about a probability of 2/21, which is about 0.095 or 9.5%.
A: Odds ratio

Q: Offense
A: It is hypothesized that the emotion of offense appears when one perceives an attempt to gain status.

Q: It is hypothesized that the emotion of [...] appears when one perceives an attempt to gain status.
A: Offense

Q: Omega
A: A hypothetical super-intelligent being used in philosophical problems. Omega is most commonly used as the predictor in Newcomb's problem. In its role as predictor, Omega's predictions occur almost certainly. In some thought experiments, Omega is also taken to be super-powerful.
Omega can be seen as analogous to Laplace's demon, or as the closest approximation to the Demon capable of existing in our universe.

Q: A hypothetical super-intelligent being used in philosophical problems. [...] is most commonly used as the predictor in Newcomb's problem. In its role as predictor, [...]'s predictions occur almost certainly. In some thought experiments, [...] is also taken to be super-powerful.
[...] can be seen as analogous to Laplace's demon, or as the closest approximation to the Demon capable of existing in our universe.
A: Omega

Q: Oops
A: There is a powerful advantage to admitting you have made a large mistake.  It's painful.  It can also change your whole life.
It is important to have the watershed moment, the moment of humbling realization.  To acknowledge a fundamental problem, not divide it into palatable bite-size mistakes.
Do not indulge in drama and become proud of admitting errors.  It is surely superior to get it right the first time.  But if you do make an error, better by far to see it all at once.  Even hedonically, it is better to take one large loss than many small ones.  The alternative is stretching out the battle with yourself over years.

Q: There is a powerful advantage to admitting you have made a large mistake.  It's painful.  It can also change your whole life.
It is important to have the watershed moment, the moment of humbling realization.  To acknowledge a fundamental problem, not divide it into palatable bite-size mistakes.
Do not indulge in drama and become proud of admitting errors.  It is surely superior to get it right the first time.  But if you do make an error, better by far to see it all at once.  Even hedonically, it is better to take one large loss than many small ones.  The alternative is stretching out the battle with yourself over years.
A: Oops

Q: Open problems on Less Wrong
A: This is a bad first draft, and may be drastically revised.  But, as a starting vision, I'm hoping these pages will do three things:
Provide a good landing page for newcomers. Let newcomers see what we’re up to, why a person might care about it, and which past posts can help them with which of their own questions.
Provide a good review, or summary of what has been discussed or figured out where, for current community members who’d like a better birds-eye picture of our project.
Provide a list of "open problems" and "articles someone should write", together with a picture of how progress on those problems would contribute to LW's project, and an index of progress to date. So that would-be authors can see useful avenues to contribute, and so that some portion of LW posts can visibly contribute to a useful, cumulative project, instead of being like a succession of randomly entertaining newspaper articles that no one much cares about afterwards. (The idea is not that  LW posts should do this, but that it would be nice if some portion did, and if LW readers ended up building some cumulative competencies over time.)
  Actual list of questions
  Questions about forming accurate beliefs:
 How accurate are peoples’ current beliefs (e.g. about themselves, about their immediate social environment or careers, and about the larger world)?
a.  What are the specific biases?
b.  To what extent do people, and people in various sub-populations, aim for accuracy in their beliefs?
c.  How accurate are most peoples’ beliefs, overall?   How accurate are the beliefs of relevant subpopulations, e.g., scientists, or avid LW-users, or people who believe they "actually try"?
How can we measure our own rationality (our own tendency to form accurate beliefs in varied domains, controlling for domain-knowledge and intelligence), and the rationality of other individuals and groups?
What practical techniques can improve the accuracy of individuals’ beliefs?
What practical techniques can improve the accuracy of groups’ beliefs?
Disagreements on Less Wrong.
Foundational questions: How is it that people can form accurate beliefs at all?  How would an ideal accurate belief-former form its beliefs?  Would such a belief-former use probabilities?  What are probabilities?  Where do priors come from?
  Questions about achieving one's goals:
To what extent does forming more accurate beliefs tend to help people achieve happiness, positive social relationships, income, longevity, actually useful philanthropy, or other goals?
It would be nice to have more closure on the "Should Rationalists Win?" question--the latest salvo here being "Extreme Rationality: It's Not That Great" which essentially answered "Not on the present state of the Art"
What techniques other than improving the accuracy of our beliefs, can help us achieve important goals?
Foundational questions:  What do humans really care about, and what formalisms can help us descriptively or normatively model human concerns?  Does it make sense to discuss value as distinct from human preferences?  How would an ideal goal-maximizer think?  What other foundations do we need, to think non-confusedly about the grand unified problem of What To Do?
See also Lesswrongwiki:Community Portal, Lesswrongwiki:Projects and Requests

Q: This is a bad first draft, and may be drastically revised.  But, as a starting vision, I'm hoping these pages will do three things:
Provide a good landing page for newcomers. Let newcomers see what we’re up to, why a person might care about it, and which past posts can help them with which of their own questions.
Provide a good review, or summary of what has been discussed or figured out where, for current community members who’d like a better birds-eye picture of our project.
Provide a list of "open problems" and "articles someone should write", together with a picture of how progress on those problems would contribute to LW's project, and an index of progress to date. So that would-be authors can see useful avenues to contribute, and so that some portion of LW posts can visibly contribute to a useful, cumulative project, instead of being like a succession of randomly entertaining newspaper articles that no one much cares about afterwards. (The idea is not that  LW posts should do this, but that it would be nice if some portion did, and if LW readers ended up building some cumulative competencies over time.)
  Actual list of questions
  Questions about forming accurate beliefs:
 How accurate are peoples’ current beliefs (e.g. about themselves, about their immediate social environment or careers, and about the larger world)?
a.  What are the specific biases?
b.  To what extent do people, and people in various sub-populations, aim for accuracy in their beliefs?
c.  How accurate are most peoples’ beliefs, overall?   How accurate are the beliefs of relevant subpopulations, e.g., scientists, or avid LW-users, or people who believe they "actually try"?
How can we measure our own rationality (our own tendency to form accurate beliefs in varied domains, controlling for domain-knowledge and intelligence), and the rationality of other individuals and groups?
What practical techniques can improve the accuracy of individuals’ beliefs?
What practical techniques can improve the accuracy of groups’ beliefs?
Disagreements on Less Wrong.
Foundational questions: How is it that people can form accurate beliefs at all?  How would an ideal accurate belief-former form its beliefs?  Would such a belief-former use probabilities?  What are probabilities?  Where do priors come from?
  Questions about achieving one's goals:
To what extent does forming more accurate beliefs tend to help people achieve happiness, positive social relationships, income, longevity, actually useful philanthropy, or other goals?
It would be nice to have more closure on the "Should Rationalists Win?" question--the latest salvo here being "Extreme Rationality: It's Not That Great" which essentially answered "Not on the present state of the Art"
What techniques other than improving the accuracy of our beliefs, can help us achieve important goals?
Foundational questions:  What do humans really care about, and what formalisms can help us descriptively or normatively model human concerns?  Does it make sense to discuss value as distinct from human preferences?  How would an ideal goal-maximizer think?  What other foundations do we need, to think non-confusedly about the grand unified problem of What To Do?
See also Lesswrongwiki:Community Portal, Lesswrongwiki:Projects and Requests
A: Open problems on Less Wrong

Q: Optimization process
A: An optimization process is a process that systematically comes up with solutions that are higher rather than lower relative to some ordering over outcomes; it hits small targets in a large search space, comes up with outcomes that you would not expect to see by random chance, atoms bumping up against each other with no direction or ordering at all. If an entity pushes reality into some state — across many contexts, not just by accident — then you could say it prefers that state.
Optimization is a very general notion that encompasses all kinds of order-generating processes other than emergence; optimization is about choosing or selecting outcomes defined as better.
Probably the optimization process you're most familiar with is that of human intelligence. Humans don't do things randomly: we have very specific goals and rearrange the world in specific ways to meet our goals. Consider the monitor on which you read these words. That monitor is a rather unlikely object to have come about by chance, and so of course, it didn't. Human economies over many years built up the infrastructure needed to build a monitor, and then built it, because people prefer to be able to see their files. It might not seem so impressive if you're used to it, but there's a lot of cognitive work that goes on behind the scenes.
Another example of an optimization process would be natural selection, notable for its "first" status if not its power or speed. Evolution works because organisms that do better at surviving and reproducing propagate more of their traits to the next generation; in this way genes with higher fitness are systematically preferred, and complex machinery bearing the strange design signature of evolved things can be built up over time.
Or consider the famous chessplaying computer, Deep Blue. Outside of the narrow domain of selecting moves for chess games, it can't do anything impressive: but as a chessplayer, it was massively more effective than virtually all humans. Humans or evolution are more domain-general optimization processes than Deep Blue, but that doesn't mean they're more effective at chess specifically. (Although note in what contexts this optimization process abstraction is useful and where it fails to be useful: it's not obvious what it would mean for "evolution" to play chess, and yet it is useful to talk about the optimization power of natural selection, or of Deep Blue.)
Measuring optimization power
One way to think mathematically about optimization, like evidence, is in information-theoretic bits. The optimization power is the amount of surprise we would have in the result if there were no optimization process present. Therefore we take the base-two logarithm of the reciprocal of the probability of the result. A one-in-a-million solution (a solution so good relative to your preference ordering that it would take a million random tries to find something that good or better) can be said to have log_2(1,000,000) = 19.9 bits of optimization. Compared to a random configuration of matter, any artifact you see is going to be much more optimized than this. The math describes laws and general principles for reasoning about optimization; as with probability theory, you oftentimes can't apply the math directly.

Q: An [...] is a process that systematically comes up with solutions that are higher rather than lower relative to some ordering over outcomes; it hits small targets in a large search space, comes up with outcomes that you would not expect to see by random chance, atoms bumping up against each other with no direction or ordering at all. If an entity pushes reality into some state — across many contexts, not just by accident — then you could say it prefers that state.
Optimization is a very general notion that encompasses all kinds of order-generating processes other than emergence; optimization is about choosing or selecting outcomes defined as better.
Probably the [...] you're most familiar with is that of human intelligence. Humans don't do things randomly: we have very specific goals and rearrange the world in specific ways to meet our goals. Consider the monitor on which you read these words. That monitor is a rather unlikely object to have come about by chance, and so of course, it didn't. Human economies over many years built up the infrastructure needed to build a monitor, and then built it, because people prefer to be able to see their files. It might not seem so impressive if you're used to it, but there's a lot of cognitive work that goes on behind the scenes.
Another example of an [...] would be natural selection, notable for its "first" status if not its power or speed. Evolution works because organisms that do better at surviving and reproducing propagate more of their traits to the next generation; in this way genes with higher fitness are systematically preferred, and complex machinery bearing the strange design signature of evolved things can be built up over time.
Or consider the famous chessplaying computer, Deep Blue. Outside of the narrow domain of selecting moves for chess games, it can't do anything impressive: but as a chessplayer, it was massively more effective than virtually all humans. Humans or evolution are more domain-general [...]es than Deep Blue, but that doesn't mean they're more effective at chess specifically. (Although note in what contexts this [...] abstraction is useful and where it fails to be useful: it's not obvious what it would mean for "evolution" to play chess, and yet it is useful to talk about the optimization power of natural selection, or of Deep Blue.)
Measuring optimization power
One way to think mathematically about optimization, like evidence, is in information-theoretic bits. The optimization power is the amount of surprise we would have in the result if there were no [...] present. Therefore we take the base-two logarithm of the reciprocal of the probability of the result. A one-in-a-million solution (a solution so good relative to your preference ordering that it would take a million random tries to find something that good or better) can be said to have log_2(1,000,000) = 19.9 bits of optimization. Compared to a random configuration of matter, any artifact you see is going to be much more optimized than this. The math describes laws and general principles for reasoning about optimization; as with probability theory, you oftentimes can't apply the math directly.
A: Optimization process

Q: Other-optimizing
A: Other-optimizing is a failure mode in which a person vastly overestimates their ability to optimize someone else's life, usually as a result of underestimating the differences between themselves and others, for example through the typical mind fallacy.

Q: [...] is a failure mode in which a person vastly overestimates their ability to optimize someone else's life, usually as a result of underestimating the differences between themselves and others, for example through the typical mind fallacy.
A: Other-optimizing

Q: Outside view
A: Taking the outside view means using an estimate based on a class of roughly similar previous cases, rather than trying to visualize the details of a process.  For example, estimating the completion time of a programming project based on how long similar projects have taken in the past, rather than by drawing up a graph of tasks and their expected completion times.  The planning fallacy is that people tend to be hugely optimistic when visualizing the details of a case, and become even more optimistic as they visualize more details.
It is controversial what exact lesson to draw from such experiments.  Robin Hanson argues that this implies that, in futurism, forecasts should be made by trying to find a reference class of similar cases, rather than by trying to visualize outcomes.  Eliezer Yudkowsky responds that this leads to "reference class tennis", and that the experiments were performed in cases where the new example was extremely similar to past examples (no more dissimilar to them than they were to each other).

Q: Taking the [...] means using an estimate based on a class of roughly similar previous cases, rather than trying to visualize the details of a process.  For example, estimating the completion time of a programming project based on how long similar projects have taken in the past, rather than by drawing up a graph of tasks and their expected completion times.  The planning fallacy is that people tend to be hugely optimistic when visualizing the details of a case, and become even more optimistic as they visualize more details.
It is controversial what exact lesson to draw from such experiments.  Robin Hanson argues that this implies that, in futurism, forecasts should be made by trying to find a reference class of similar cases, rather than by trying to visualize outcomes.  Eliezer Yudkowsky responds that this leads to "reference class tennis", and that the experiments were performed in cases where the new example was extremely similar to past examples (no more dissimilar to them than they were to each other).
A: Outside view

Q: Overcoming Bias
A: Overcoming Bias is a group blog on the systemic mistakes humans make, and how we can possibly correct them. The primary contributors are Robin Hanson of George Mason University and Eliezer Yudkowsky of the Singularity Institute. Common topics include "cognitive psychology, evolutionary psychology, microeconomics, applied statistics, social psychology, probability and decision theory, even a bit of Artificial Intelligence now and then."
The complete list of Yudkowsky's posts on Overcoming Bias was compiled by Andrew Hay here.
A categorized list of Yudkowsky's posts moved from Overcoming Bias to Less Wrong is available at Less Wrong/All Articles.

Q: [...] is a group blog on the systemic mistakes humans make, and how we can possibly correct them. The primary contributors are Robin Hanson of George Mason University and Eliezer Yudkowsky of the Singularity Institute. Common topics include "cognitive psychology, evolutionary psychology, microeconomics, applied statistics, social psychology, probability and decision theory, even a bit of Artificial Intelligence now and then."
The complete list of Yudkowsky's posts on [...] was compiled by Andrew Hay here.
A categorized list of Yudkowsky's posts moved from [...] to Less Wrong is available at Less Wrong/All Articles.
A: Overcoming Bias

Q: Overconfidence
A: Overconfidence is the state of being more certain than is justified, given your priors and the evidence available.

Q: [...] is the state of being more certain than is justified, given your priors and the evidence available.
A: Overconfidence

Q: Philosophical zombie
A: A philosophical zombie or p-zombie is a hypothetical entity that looks and behaves exactly like a human (often stipulated to be atom-by-atom identical to a human) but is not actually conscious.  A p-zombie is as likely as anyone else to ask, "When I see red, do I see the same color that you see when you see red?" but they have no real experience of the color red; the zombie's speech can be explained purely in mechanistic terms. The zombie thought experiment is purported to show that consciousness cannot be reduced to merely physical things: our universe is purported to perhaps have special "bridging laws" which evoke a mind into existence when there are atoms in a suitable brain-like configuration.
Critics deny the possibility of zombies: if a p-zombie is atom-by-atom identical to a human being in our universe, then our speech can be explained by the same mechanisms as the zombie's, and yet it would seem awfully peculiar that our words and actions would have one entirely materialistic explanation, but also, furthermore, our universe happens to contain exactly the right bridging law such that our experiences are meaningful and our consciousness syncs up with what our merely physical bodies do. It's too much of a stretch: Occam's razor dictates that we favor a monistic universe with one uniform set of laws.

Q: A [...] or p-zombie is a hypothetical entity that looks and behaves exactly like a human (often stipulated to be atom-by-atom identical to a human) but is not actually conscious.  A p-zombie is as likely as anyone else to ask, "When I see red, do I see the same color that you see when you see red?" but they have no real experience of the color red; the zombie's speech can be explained purely in mechanistic terms. The zombie thought experiment is purported to show that consciousness cannot be reduced to merely physical things: our universe is purported to perhaps have special "bridging laws" which evoke a mind into existence when there are atoms in a suitable brain-like configuration.
Critics deny the possibility of zombies: if a p-zombie is atom-by-atom identical to a human being in our universe, then our speech can be explained by the same mechanisms as the zombie's, and yet it would seem awfully peculiar that our words and actions would have one entirely materialistic explanation, but also, furthermore, our universe happens to contain exactly the right bridging law such that our experiences are meaningful and our consciousness syncs up with what our merely physical bodies do. It's too much of a stretch: Occam's razor dictates that we favor a monistic universe with one uniform set of laws.
A: Philosophical zombie

Q: Paperclip maximizer
A: The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.
—Eliezer Yudkowsky, Artificial Intelligence as a Positive and Negative Factor in Global Risk
The paperclip maximizer is the canonical thought experiment showing how an artificial general intelligence, even one with an apparently innocuous and impractical goal, would ultimately destroy humanity--unless its goal is the preservation of human values.  It illustrates the contingency of human values: An extremely powerful optimizer (a highly intelligent agent) could seek goals that are completely alien to ours, and as a side-effect destroy us by consuming resources essential to our survival.
Contents
1 Description
2 Conclusions
3 Similar thought experiments
4 External links
5 References
6 Blog posts
7 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Description
First described by Bostrom (2003), the paperclip maximizer is an artificial general intelligence whose goal is to maximize the number of paperclips in its collection. If it has  been constructed with a roughly human level of general intelligence, the AGI might collect paperclips, earn money to  buy paperclips, or  begin to manufacture paperclips.
Most importantly, however, it would undergo an intelligence explosion: It would work to improve its own intelligence, where "intelligence" is understood in the sense of  optimization power, the ability to maximize a reward/utility function--in this case, the number of paperclips. The AGI would do so, not because it would value more intelligence in its own right, but because more intelligence would help it achieve its goal. Having done so, it would produce more paperclips, and also used its enhanced intelligence to further improve  its own intelligence. Continuing this process, it would undergo an Intelligence explosion  and reach far-above-human levels.
At this point, it would innovate new techniques to maximize the number of paperclips. Ultimately, it would convert all the mass of the Earth or the solar system to paperclips.
This may seem more like super-stupidity than super-intelligence. For humans, it would indeed be stupidity, as it would constitute failure to fulfill many of our important terminal values, such as life, love, and variety.  But the AGI under consideration has a goal system very different from humans. It has the one, simple goal of maximizing the number of paperclips, and human life, learning, joy, and so on are not specified as goals. The AGI is simply an optimization process--a goal-seeker, a utility-function-maximizer. Its values can be completely alien to ours. If its utility function is to maximize paperclips, then unless it is buggy, it will do exactly that.
An intelligence explosion  is not necessary to a paperclipping scenario. If society keeps getting increasingly automated and AI-dominated, then the first borderline AGI might manage to take over the rest using some relatively narrow-domain trick, that doesn't  require very high general intelligence.
Conclusions
The paperclip maximizer illustrates the arbitrariness and contingency of human values. An entity can be an optimizer without sharing any of the complex mix of human terminal values, which developed under the particular selection pressures found in our environment of evolutionary adaptation.
Any future AGI must be built to specifically optimize for human values as its terminal value (goal).Human values don't spontaneously emerge in a generic optimization process.
Thus, if an AGI is not specifically programmed to be benevolent to humans, it will be almost as  dangerous as if it were designed to be malevolent.
Similar thought experiments
Paperclip maximization was chosen as an example because it is unlikely to be implemented in AGIs, and has little apparent danger or emotional load (in contrast to, for example, curing cancer, winning wars).
Other goals for AGIs have been used to illustrate similar concepts.
Nick Hay used stamp collection  as an example of a morally neutral goal.
If the  machine's terminal value is to make a pure calculation like solving the Riemann Hypothesis  would convert all available mass to computronium (the most efficient possible computer processors).
Even a machine with a goal that apparently supports human terminal values would produce similar outcomes unless it had the full complement of human values as its goal. For example, an AGI whose terminal value is to increase the number of smiles, as a proxy for human happiness, would tile the solar system with smiley faces (Yudkowsky 2008).

Q: The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.
—Eliezer Yudkowsky, Artificial Intelligence as a Positive and Negative Factor in Global Risk
The [...] is the canonical thought experiment showing how an artificial general intelligence, even one with an apparently innocuous and impractical goal, would ultimately destroy humanity--unless its goal is the preservation of human values.  It illustrates the contingency of human values: An extremely powerful optimizer (a highly intelligent agent) could seek goals that are completely alien to ours, and as a side-effect destroy us by consuming resources essential to our survival.
Contents
1 Description
2 Conclusions
3 Similar thought experiments
4 External links
5 References
6 Blog posts
7 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Description
First described by Bostrom (2003), the [...] is an artificial general intelligence whose goal is to maximize the number of paperclips in its collection. If it has  been constructed with a roughly human level of general intelligence, the AGI might collect paperclips, earn money to  buy paperclips, or  begin to manufacture paperclips.
Most importantly, however, it would undergo an intelligence explosion: It would work to improve its own intelligence, where "intelligence" is understood in the sense of  optimization power, the ability to maximize a reward/utility function--in this case, the number of paperclips. The AGI would do so, not because it would value more intelligence in its own right, but because more intelligence would help it achieve its goal. Having done so, it would produce more paperclips, and also used its enhanced intelligence to further improve  its own intelligence. Continuing this process, it would undergo an Intelligence explosion  and reach far-above-human levels.
At this point, it would innovate new techniques to maximize the number of paperclips. Ultimately, it would convert all the mass of the Earth or the solar system to paperclips.
This may seem more like super-stupidity than super-intelligence. For humans, it would indeed be stupidity, as it would constitute failure to fulfill many of our important terminal values, such as life, love, and variety.  But the AGI under consideration has a goal system very different from humans. It has the one, simple goal of maximizing the number of paperclips, and human life, learning, joy, and so on are not specified as goals. The AGI is simply an optimization process--a goal-seeker, a utility-function-maximizer. Its values can be completely alien to ours. If its utility function is to maximize paperclips, then unless it is buggy, it will do exactly that.
An intelligence explosion  is not necessary to a paperclipping scenario. If society keeps getting increasingly automated and AI-dominated, then the first borderline AGI might manage to take over the rest using some relatively narrow-domain trick, that doesn't  require very high general intelligence.
Conclusions
The [...] illustrates the arbitrariness and contingency of human values. An entity can be an optimizer without sharing any of the complex mix of human terminal values, which developed under the particular selection pressures found in our environment of evolutionary adaptation.
Any future AGI must be built to specifically optimize for human values as its terminal value (goal).Human values don't spontaneously emerge in a generic optimization process.
Thus, if an AGI is not specifically programmed to be benevolent to humans, it will be almost as  dangerous as if it were designed to be malevolent.
Similar thought experiments
Paperclip maximization was chosen as an example because it is unlikely to be implemented in AGIs, and has little apparent danger or emotional load (in contrast to, for example, curing cancer, winning wars).
Other goals for AGIs have been used to illustrate similar concepts.
Nick Hay used stamp collection  as an example of a morally neutral goal.
If the  machine's terminal value is to make a pure calculation like solving the Riemann Hypothesis  would convert all available mass to computronium (the most efficient possible computer processors).
Even a machine with a goal that apparently supports human terminal values would produce similar outcomes unless it had the full complement of human values as its goal. For example, an AGI whose terminal value is to increase the number of smiles, as a proxy for human happiness, would tile the solar system with smiley faces (Yudkowsky 2008).
A: Paperclip maximizer

Q: Paranoid debating
A: A variant of The Aumann Game where one player purposefully subverts the group estimate. Similar to The Aumann Game, the activity consists of a group jointly producing a confidence interval for an unknown, but verifiable quantity, which is then scored for accuracy and calibration. One individual is designated the spokesperson, who is responsible for choosing the final estimate. However, before the activity begins, one individual is secretly assigned the role of misleading the other members. The deceiver is scored higher the worse the final estimate is.
The activity is intended to teach accurate estimate, proper agreement techniques, and recognition of deception.
A typical subject for the game might be "How much maize is produced in Mexico annually?".
Contents
1 Rules
1.1 Simplest variant
1.2 Advocate variant, #1
1.3 Advocate variant, #2
1.4 Variation-by-argument variant
1.5 Southern California Variant #1
2 Questions
3 Scoring
4 Blog posts
5 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Rules
Select player roles. In person, each player receives or selects a card from a pack of role cards. For 4 players, create a pack of role cards by combining 3 black cards with 1 red card. For 4-6 players there should be 1 red card and the rest black with the rest being enough for one card per person. For 7-9 players, 2 red cards. Some variants include a role named the Advocate, which you can designate one of the black cards to represent.
Simplest variant
Each player receives a role. No advocate.
A question is asked.
Players discuss for 20 minutes, then write down their individual response on a card.
The answer is researched.
Scores are assigned.
Advocate variant, #1
Each player receives a role. One advocate in the deck. The player who receives the Advocate displays it to the group.
A question is asked.
Players discuss for 20 minutes, attempting to convince the advocate. The advocate writes down their response on a card. This is the group's answer.
The answer is researched, scores are assigned.
Advocate variant, #2
Each player receives a role. One advocate in the deck. No player may display their card.
A question is asked.
Players discuss for 20 minutes. Anyone may say anything. At the end, the advocate writes down what they think the group's response is on a card, and the group is scored for this.
Answer researched, scores assigned.
Variation-by-argument variant
Each player receives a role. No advocate. No player may display their card.
A question is asked.
Players have 2-5 minutes to write down their initial, individual estimate.
Players discuss for 20 minutes. Anyone may say anything. At the end, players write their revised estimates on their card.
Players are scored based on their delta -- the more you go toward the correct answer from your initial estimate, the more points.
Southern California Variant #1
At the February 2011 Southern California LW Meetup we tried playing the game.  For questions we bought a game of Wits & Wagers (which has trivia questions with numerical answers) and looked at the cards to find questions that were about substantive topics where Fermi estimates seemed useful.  The speaker/advocate was chosen on a rotating basis so that everyone gets at least one chance to play that role, and cards are dealt from a deck of playing cards to everyone else.  Red cards mean you're trying to make the group deliver a bad answer.  Black cards mean you're trying to make the group deliver a good answer.  This makes the number of people to be suspicious of itself an unknown parameter and leads to funny outcomes and interesting coordination problems.  Scoring used the experimental scoring code that is intended to assign the most credit to small error bars around high confidence correct answers.
Questions
It's really easy to ask a question that is then very difficult to answer later. For example, the question "How many miles of railroad are there in Africa?" is somewhat difficult to answer. Walking through the CIA World Fact Book one country at a time, we arrived at an answer in the range of 48,000-49,000. However, in cross-checking that information, we discovered that in Uganda, there are only 125 miles of active railroad, but 1200km listed in the Fact Book. It seems likely, therefore, that the total estimate includes some non-active miles of railroad, and is thus too high. This section is here to list good and bad questions and resources to get questions from or answer questions unusually easily. If listing an answer, please make the text of the answer white so people can use it if they want.
Scoring
A not-so-trivial inconvenience to playing the game is figuring out how to score it properly.
To make this easier there is now a tentative file format for representing a game of paranoid debate and a python script for scoring games represented in this format.  If you'd like to download or edit this software check out this github project.  Please note that the game format and the code are very likely to evolve to remove bugs and support whatever sort of play turns out to be the most fun and/or educational.

Q: A variant of The Aumann Game where one player purposefully subverts the group estimate. Similar to The Aumann Game, the activity consists of a group jointly producing a confidence interval for an unknown, but verifiable quantity, which is then scored for accuracy and calibration. One individual is designated the spokesperson, who is responsible for choosing the final estimate. However, before the activity begins, one individual is secretly assigned the role of misleading the other members. The deceiver is scored higher the worse the final estimate is.
The activity is intended to teach accurate estimate, proper agreement techniques, and recognition of deception.
A typical subject for the game might be "How much maize is produced in Mexico annually?".
Contents
1 Rules
1.1 Simplest variant
1.2 Advocate variant, #1
1.3 Advocate variant, #2
1.4 Variation-by-argument variant
1.5 Southern California Variant #1
2 Questions
3 Scoring
4 Blog posts
5 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Rules
Select player roles. In person, each player receives or selects a card from a pack of role cards. For 4 players, create a pack of role cards by combining 3 black cards with 1 red card. For 4-6 players there should be 1 red card and the rest black with the rest being enough for one card per person. For 7-9 players, 2 red cards. Some variants include a role named the Advocate, which you can designate one of the black cards to represent.
Simplest variant
Each player receives a role. No advocate.
A question is asked.
Players discuss for 20 minutes, then write down their individual response on a card.
The answer is researched.
Scores are assigned.
Advocate variant, #1
Each player receives a role. One advocate in the deck. The player who receives the Advocate displays it to the group.
A question is asked.
Players discuss for 20 minutes, attempting to convince the advocate. The advocate writes down their response on a card. This is the group's answer.
The answer is researched, scores are assigned.
Advocate variant, #2
Each player receives a role. One advocate in the deck. No player may display their card.
A question is asked.
Players discuss for 20 minutes. Anyone may say anything. At the end, the advocate writes down what they think the group's response is on a card, and the group is scored for this.
Answer researched, scores assigned.
Variation-by-argument variant
Each player receives a role. No advocate. No player may display their card.
A question is asked.
Players have 2-5 minutes to write down their initial, individual estimate.
Players discuss for 20 minutes. Anyone may say anything. At the end, players write their revised estimates on their card.
Players are scored based on their delta -- the more you go toward the correct answer from your initial estimate, the more points.
Southern California Variant #1
At the February 2011 Southern California LW Meetup we tried playing the game.  For questions we bought a game of Wits & Wagers (which has trivia questions with numerical answers) and looked at the cards to find questions that were about substantive topics where Fermi estimates seemed useful.  The speaker/advocate was chosen on a rotating basis so that everyone gets at least one chance to play that role, and cards are dealt from a deck of playing cards to everyone else.  Red cards mean you're trying to make the group deliver a bad answer.  Black cards mean you're trying to make the group deliver a good answer.  This makes the number of people to be suspicious of itself an unknown parameter and leads to funny outcomes and interesting coordination problems.  Scoring used the experimental scoring code that is intended to assign the most credit to small error bars around high confidence correct answers.
Questions
It's really easy to ask a question that is then very difficult to answer later. For example, the question "How many miles of railroad are there in Africa?" is somewhat difficult to answer. Walking through the CIA World Fact Book one country at a time, we arrived at an answer in the range of 48,000-49,000. However, in cross-checking that information, we discovered that in Uganda, there are only 125 miles of active railroad, but 1200km listed in the Fact Book. It seems likely, therefore, that the total estimate includes some non-active miles of railroad, and is thus too high. This section is here to list good and bad questions and resources to get questions from or answer questions unusually easily. If listing an answer, please make the text of the answer white so people can use it if they want.
Scoring
A not-so-trivial inconvenience to playing the game is figuring out how to score it properly.
To make this easier there is now a tentative file format for representing a game of paranoid debate and a python script for scoring games represented in this format.  If you'd like to download or edit this software check out this github project.  Please note that the game format and the code are very likely to evolve to remove bugs and support whatever sort of play turns out to be the most fun and/or educational.
A: Paranoid debating

Q: Pascal\'s mugging
A: Pascal's mugging refers to a thought experiment in decision theory, a finite analogue of Pascal's wager.  The situation is dramatized by a mugger:
Now suppose someone comes to me and says, "Give me five dollars, or I'll use my magic powers from outside the Matrix to run a Turing machine that simulates and kills 3^^^^3 people."
—Pascal's Mugging: Tiny Probabilities of Vast Utilities
If an agent's utilities over outcomes can potentially grow much faster than the probability of those outcomes diminishes, then it will be dominated by tiny probabilities of hugely important outcomes.  The prior over computable universes in Solomonoff induction seems to have this problem in particular - more generally, if prior probability goes as simplicity of physical law, then small increases in complexity can correspond to enormous increases in the size of even a finite universe.
Intuitively, one is not inclined to acquiesce to the mugger's demands - or even pay all that much attention one way or another - but what kind of prior does this imply?
Robin Hanson has suggested penalizing the prior probability of hypotheses which argue that we are in a surprisingly unique position to affect large numbers of other people who cannot symmetrically affect us.  Since only one in 3^^^^3 people can be in a unique position to ordain the existence of at least 3^^^^3 other people who are not symmetrically in such a situation themselves, the prior probability would be penalized by a factor on the same order as the utility.
Peter de Blanc has proven that if an agent assigns a finite probability to all computable hypotheses (which probability may be computable, or computably bounded below) and assigns unboundedly large finite utilities over percept sequences (that is, the utilities are a direct function of sensory information, not over possible external universes that could be responsible for the sensory percepts; and these utilities are computable, or computably bounded below) then the sum in the expected utility formula does not converge.
Peter de Blanc's paper, and the Pascal's Mugging argument, are sometimes misinterpreted as showing that any agent with an unbounded finite utility function over outcomes is not consistent, but this has yet to be demonstrated.

Q: [...] refers to a thought experiment in decision theory, a finite analogue of Pascal's wager.  The situation is dramatized by a mugger:
Now suppose someone comes to me and says, "Give me five dollars, or I'll use my magic powers from outside the Matrix to run a Turing machine that simulates and kills 3^^^^3 people."
—Pascal's Mugging: Tiny Probabilities of Vast Utilities
If an agent's utilities over outcomes can potentially grow much faster than the probability of those outcomes diminishes, then it will be dominated by tiny probabilities of hugely important outcomes.  The prior over computable universes in Solomonoff induction seems to have this problem in particular - more generally, if prior probability goes as simplicity of physical law, then small increases in complexity can correspond to enormous increases in the size of even a finite universe.
Intuitively, one is not inclined to acquiesce to the mugger's demands - or even pay all that much attention one way or another - but what kind of prior does this imply?
Robin Hanson has suggested penalizing the prior probability of hypotheses which argue that we are in a surprisingly unique position to affect large numbers of other people who cannot symmetrically affect us.  Since only one in 3^^^^3 people can be in a unique position to ordain the existence of at least 3^^^^3 other people who are not symmetrically in such a situation themselves, the prior probability would be penalized by a factor on the same order as the utility.
Peter de Blanc has proven that if an agent assigns a finite probability to all computable hypotheses (which probability may be computable, or computably bounded below) and assigns unboundedly large finite utilities over percept sequences (that is, the utilities are a direct function of sensory information, not over possible external universes that could be responsible for the sensory percepts; and these utilities are computable, or computably bounded below) then the sum in the expected utility formula does not converge.
Peter de Blanc's paper, and the Pascal's Mugging argument, are sometimes misinterpreted as showing that any agent with an unbounded finite utility function over outcomes is not consistent, but this has yet to be demonstrated.
A: Pascal\'s mugging

Q: Peak-end rule
A: According to the peak-end rule, we judge our past experiences almost entirely on how they were at their peak (pleasant or unpleasant) and how they ended. The rule states that we do not judge our experiences on the net pleasantness of unpleasantness or on how long the experience lasted.

Q: According to the [...], we judge our past experiences almost entirely on how they were at their peak (pleasant or unpleasant) and how they ended. The rule states that we do not judge our experiences on the net pleasantness of unpleasantness or on how long the experience lasted.
A: Peak-end rule

Q: Perceptual control theory
A: Perceptual control theory (PCT) is a psychological theory of animal and human behavior. PCT postulates that an organism's behavior is a means of controlling its perceptions.
Controversy
It's unclear whether PCT is a valid theory that explains anything. The advocates of the theory on the blog were unable to make a clear case for it. For example, see these critical comments about the paper version of this technical report.

Q: [...] (PCT) is a psychological theory of animal and human behavior. PCT postulates that an organism's behavior is a means of controlling its perceptions.
Controversy
It's unclear whether PCT is a valid theory that explains anything. The advocates of the theory on the blog were unable to make a clear case for it. For example, see these critical comments about the paper version of this technical report.
A: Perceptual control theory

Q: Phlogiston
A: Phlogiston was the 18 century's answer to the Elemental Fire of the Greek alchemists.  Ignite wood, and let it burn.  What is the orangey-bright "fire" stuff?  Why does the wood transform into ash?  To both questions, the 18th-century chemists answered, "phlogiston".
...and that was it, you see, that was their answer:  "Phlogiston."

Q: [...] was the 18 century's answer to the Elemental Fire of the Greek alchemists.  Ignite wood, and let it burn.  What is the orangey-bright "fire" stuff?  Why does the wood transform into ash?  To both questions, the 18th-century chemists answered, "[...]".
...and that was it, you see, that was their answer:  "[...]."
A: Phlogiston

Q: Planning fallacy
A: It always takes longer than you expect, even when you take into account Hofstadter's Law.
—Hofstadter's Law, from Gödel, Escher, Bach: An Eternal Golden Braid
A common cognitive bias resulting in predicting absurdly short timeframes for planned projects, famously observed with, among other projects, the  Sydney Opera House, completed ten years late and a hundred million dollars overbudget.
Symptomatic of the Planning Fallacy is an assumption of a best-case scenario; people plan as if everything will go smoothly, as hoped for, with no unexpected delays. In practice, this is typically not the case, and delays quickly mount.
The bias also seems to be related to taking an "inside", detail-oriented view of the project to be planned; studies show that the more detailed a plan is, the more optimistically inaccurate it is likely to be.
Debiasing techniques
When possible, take the outside view. Avoid estimating the time for a project by adding time estimates for sub-tasks; instead, look for previously completed projects of similar type and scale, and base the estimate on how long those other projects took.

Q: It always takes longer than you expect, even when you take into account Hofstadter's Law.
—Hofstadter's Law, from Gödel, Escher, Bach: An Eternal Golden Braid
A common cognitive bias resulting in predicting absurdly short timeframes for planned projects, famously observed with, among other projects, the  Sydney Opera House, completed ten years late and a hundred million dollars overbudget.
Symptomatic of the Planning Fallacy is an assumption of a best-case scenario; people plan as if everything will go smoothly, as hoped for, with no unexpected delays. In practice, this is typically not the case, and delays quickly mount.
The bias also seems to be related to taking an "inside", detail-oriented view of the project to be planned; studies show that the more detailed a plan is, the more optimistically inaccurate it is likely to be.
Debiasing techniques
When possible, take the outside view. Avoid estimating the time for a project by adding time estimates for sub-tasks; instead, look for previously completed projects of similar type and scale, and base the estimate on how long those other projects took.
A: Planning fallacy

Q: Playing to win
A: Playing to Win: Becoming the Champion is a book written by David Sirlin with the purpose of "teach[ing] you to win."  The article was linked by CronoDAS's LessWrong article "Playing to Win".  The term "playing to win" comes from Sirlin's book and can be described as using every means necessary to win as long as those means are legal within the structure of the game being played.
The first step in becoming a top player is the realization that playing to win means doing whatever most increases your chances of winning. That is true by definition of playing to win. The game knows no rules of “honor” or of “cheapness.” The game only knows winning and losing.  (Emphasis original.)
Sirlin also shows a contrast between people who play to win and scrubs who do not.
The scrub [...] is bound up by an intricate construct of fictitious rules that prevents him from ever truly competing.
An example of these fictitious rules can be found in the video game Street Fighter.  Scrubs often consider repeating the same moves over and over as a "cheap" or unworthy tactic and will refuse to use them even though they are legitimate strategies.

Q: Playing to Win: Becoming the Champion is a book written by David Sirlin with the purpose of "teach[ing] you to win."  The article was linked by CronoDAS's LessWrong article "Playing to Win".  The term "[...]" comes from Sirlin's book and can be described as using every means necessary to win as long as those means are legal within the structure of the game being played.
The first step in becoming a top player is the realization that [...] means doing whatever most increases your chances of winning. That is true by definition of [...]. The game knows no rules of “honor” or of “cheapness.” The game only knows winning and losing.  (Emphasis original.)
Sirlin also shows a contrast between people who play to win and scrubs who do not.
The scrub [...] is bound up by an intricate construct of fictitious rules that prevents him from ever truly competing.
An example of these fictitious rules can be found in the video game Street Fighter.  Scrubs often consider repeating the same moves over and over as a "cheap" or unworthy tactic and will refuse to use them even though they are legitimate strategies.
A: Playing to win

Q: Policy debates should not appear one-sided
A: Almost any course of action has both costs and benefits: optimizing for one value means trading off another. To show both faces of the issue, policy debates shouldn't appear one-sided.

Q: Almost any course of action has both costs and benefits: optimizing for one value means trading off another. To show both faces of the issue, policy debates shouldn't appear one-sided.
A: Policy debates should not appear one-sided

Q: Politics is the Mind-Killer
A: People go funny in the head when talking about politics.  The evolutionary reasons for this are so obvious as to be worth belaboring:  In the ancestral environment, politics was a matter of life and death.  And sex, and wealth, and allies, and reputation...  When, today, you get into an argument about whether "we" ought to raise the minimum wage, you're executing adaptations for an ancestral environment where being on the wrong side of the argument could get you killed...  Politics is an extension of war by other means.  Arguments are soldiers.  Once you know which side you're on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it's like stabbing your soldiers in the back - providing aid and comfort to the enemy.
—Eliezer Yudkowsky, Politics is the Mind-Killer
A sequence on the various ways that politics damages our sanity - including, of course, making it harder to change our minds on political issues.
This page is about Eliezer Yudkowsky's sequence.  For ongoing discussion of these issues at LW Wiki, see Mind-killer and Color politics.
Blog posts
A Fable of Science and Politics
Politics is the Mind-Killer
Policy Debates Should Not Appear One-Sided
The Scales of Justice, the Notebook of Rationality
Correspondence Bias
Are Your Enemies Innately Evil?
The Robbers Cave Experiment
Reversed Stupidity is Not Intelligence
Argument Screens Off Authority
Hug the Query
Rationality and the English Language
Politics and Awful Art (but first read: The Litany Against Gurus)
False Laughter
Human Evil and Muddled Thinking
See also
Closely related to Death Spirals and the Cult Attractor, particularly:

Q: People go funny in the head when talking about politics.  The evolutionary reasons for this are so obvious as to be worth belaboring:  In the ancestral environment, politics was a matter of life and death.  And sex, and wealth, and allies, and reputation...  When, today, you get into an argument about whether "we" ought to raise the minimum wage, you're executing adaptations for an ancestral environment where being on the wrong side of the argument could get you killed...  Politics is an extension of war by other means.  Arguments are soldiers.  Once you know which side you're on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it's like stabbing your soldiers in the back - providing aid and comfort to the enemy.
—Eliezer Yudkowsky, [...]
A sequence on the various ways that politics damages our sanity - including, of course, making it harder to change our minds on political issues.
This page is about Eliezer Yudkowsky's sequence.  For ongoing discussion of these issues at LW Wiki, see Mind-killer and Color politics.
Blog posts
A Fable of Science and Politics
[...]
Policy Debates Should Not Appear One-Sided
The Scales of Justice, the Notebook of Rationality
Correspondence Bias
Are Your Enemies Innately Evil?
The Robbers Cave Experiment
Reversed Stupidity is Not Intelligence
Argument Screens Off Authority
Hug the Query
Rationality and the English Language
Politics and Awful Art (but first read: The Litany Against Gurus)
False Laughter
Human Evil and Muddled Thinking
See also
Closely related to Death Spirals and the Cult Attractor, particularly:
A: Politics is the Mind-Killer

Q: Positive bias
A: Positive or confirmation bias is a tendency to test hypotheses with positive rather than negative examples, thus risking to miss obvious disconfirming tests.
Kevin Kelly argues that negative results should be "saved, shared, compiled and analyzed, instead of being dumped. Positive results may increase their credibility when linked to negative results." [1]  If so then this bias is particularly dangerous as the lack of negative results would themselves cast doubt on even entirely valid conclusions.
As an extreme example, imagine one hundred algorithms for stock market prediction placed in one hundred safety deposit boxes under a hundred different assumed names.  Ten years later, to great fanfare, only one box, the one containing the most accurate post-facto results, is opened.  There is no universally accepted and undeniable way to prove that any of the other 99 did or didn't exist without a discipline that forces negative result reporting.  Even a person who had filed away only one algorithm under their own name, once, would therefore be suspect.

Q: Positive or confirmation bias is a tendency to test hypotheses with positive rather than negative examples, thus risking to miss obvious disconfirming tests.
Kevin Kelly argues that negative results should be "saved, shared, compiled and analyzed, instead of being dumped. Positive results may increase their credibility when linked to negative results." [1]  If so then this bias is particularly dangerous as the lack of negative results would themselves cast doubt on even entirely valid conclusions.
As an extreme example, imagine one hundred algorithms for stock market prediction placed in one hundred safety deposit boxes under a hundred different assumed names.  Ten years later, to great fanfare, only one box, the one containing the most accurate post-facto results, is opened.  There is no universally accepted and undeniable way to prove that any of the other 99 did or didn't exist without a discipline that forces negative result reporting.  Even a person who had filed away only one algorithm under their own name, once, would therefore be suspect.
A: Positive bias

Q: Positivism, Self Deception, and Neuroscience (sequence)
A: A sequence of posts by Yvain.

Q: A sequence of posts by Yvain.
A: Positivism, Self Deception, and Neuroscience (sequence)

Q: Possibility
A: Taken literally, saying that there is a "possibility" of infection tells me nothing.  The probability could be 1/1,000,000,000,000 and it would still be technically correct to describe the outcome as "possible".
—Useless Medical Disclaimers
Words in natural language carry connotations that may become misleading when the words get applied with technical precision. While it's not technically a lie to say that it's possible to win a lottery, the statement is deceptive. It's much more precise, for communication of the actual fact through connotation, to say that it's impossible to win the lottery. This is an example of antiprediction.

Q: Taken literally, saying that there is a "[...]" of infection tells me nothing.  The probability could be 1/1,000,000,000,000 and it would still be technically correct to describe the outcome as "possible".
—Useless Medical Disclaimers
Words in natural language carry connotations that may become misleading when the words get applied with technical precision. While it's not technically a lie to say that it's possible to win a lottery, the statement is deceptive. It's much more precise, for communication of the actual fact through connotation, to say that it's impossible to win the lottery. This is an example of antiprediction.
A: Possibility

Q: Possible world
A: In philosophy, a possible world is one that is internally consistent, even if it is counterfactual.

Q: In philosophy, a [...] is one that is internally consistent, even if it is counterfactual.
A: Possible world

Q: Prediction
A: A prediction is a statement or claim that a particular event will occur in the future in more certain terms than a forecast.
Blog posts
Futuristic Predictions as Consumable Goods
See also
Antiprediction
Making beliefs pay rent
Prediction market
Forecast
Calibration
Black swan
External links
PredictionBook, a website that keeps track of predictions and calibration levels

Q: A [...] is a statement or claim that a particular event will occur in the future in more certain terms than a forecast.
Blog posts
Futuristic [...]s as Consumable Goods
See also
Anti[...]
Making beliefs pay rent
[...] market
Forecast
Calibration
Black swan
External links
[...]Book, a website that keeps track of [...]s and calibration levels
A: Prediction

Q: Prediction market
A: Prediction markets are speculative markets created for the purpose of making predictions. Assets are created whose final cash value is tied to a particular event or parameter. The current market prices can then be interpreted as predictions of the probability of the event or the expected value of the parameter. Prediction markets are thus structured as betting exchanges, without any risk for the bookmaker. Robin Hanson has suggested many innovative additions to predictions markets such as conditional predictions.
People who buy low and sell high are rewarded for improving the market prediction, while those who buy high and sell low are punished for degrading the market prediction. Evidence so far suggests that prediction markets are at least as accurate as other institutions predicting the same events with a similar pool of participants.
Predictions markets have been used by organizations such as Google, General Electric, and Microsoft; several online and commercial prediction markets are also in operation. Historically, prediction markets have often been used to predict election outcomes.

Q: [...]s are speculative markets created for the purpose of making predictions. Assets are created whose final cash value is tied to a particular event or parameter. The current market prices can then be interpreted as predictions of the probability of the event or the expected value of the parameter. [...]s are thus structured as betting exchanges, without any risk for the bookmaker. Robin Hanson has suggested many innovative additions to predictions markets such as conditional predictions.
People who buy low and sell high are rewarded for improving the market prediction, while those who buy high and sell low are punished for degrading the market prediction. Evidence so far suggests that [...]s are at least as accurate as other institutions predicting the same events with a similar pool of participants.
Predictions markets have been used by organizations such as Google, General Electric, and Microsoft; several online and commercial [...]s are also in operation. Historically, [...]s have often been used to predict election outcomes.
A: Prediction market

Q: Preference
A: Preference is a normative side of optimization. Preference is roughly equivalent to goals and values, but the concept refers more to the sum total of all agent's goals and dispositions than to the individual components. For example, for an agent that runs a decision theory based on expected utility maximization, preference should specify both prior and utility function.

Q: [...] is a normative side of optimization. [...] is roughly equivalent to goals and values, but the concept refers more to the sum total of all agent's goals and dispositions than to the individual components. For example, for an agent that runs a decision theory based on expected utility maximization, [...] should specify both prior and utility function.
A: Preference

Q: Priming
A: Priming is a psychological phenomenon that consists in early stimulus influencing later thoughts and behavior.

Q: [...] is a psychological phenomenon that consists in early stimulus influencing later thoughts and behavior.
A: Priming

Q: Priors
A: A Bayesian uses Bayes's Theorem to  update beliefs based on the evidence.  This requires that, even in advance of seeing the evidence, you have beliefs about what the evidence means - how likely you are to see the evidence, if various hypotheses are true - and how likely those hypotheses were, in advance of seeing the evidence.  To calculate a posterior probability using Bayes's Theorem, you need a prior probability and likelihood distribution.
Suppose you had a barrel containing some number of red and white balls.  If you start with the belief that each ball was independently assigned red color (vs. white color) at some fixed probability between 0 and 1, and you start out ignorant of this fixed probability (the parameter could anywhere between 0 and 1), then each red ball you see makes it more likely that the next ball will be red.  (By Laplace's Rule of Succession.)
On the other hand, if you start out with the prior belief that the barrel contains exactly 10 red balls and 10 white balls, then each red ball you see makes it less likely that the next ball will be red (because there are fewer red balls remaining).
Thus our prior can affect how we interpret the evidence.  The first prior is an inductive prior; things that happened before are predicted to happen again with greater probability.  The second prior is anti-inductive; the more red balls we see, the fewer we expect to see in the future.
In both cases, you started out believing something about the barrel - presumably because someone else told you, or because you saw it with your own eyes.  But then their words, or even your own eyesight, was evidence, and you must have had prior beliefs about probabilities and likelihoods in order to interpret the evidence.  So it seems that an ideal Bayesian would need some sort of inductive prior at the very moment they were born; and where an ideal Bayesian would get this prior, has occasionally been a matter of considerable controversy in the philosophy of probability.
The phrase  "Prior probability" (as opposed to "priors") usually refers to a point estimate already based on considerable evidence - for example when we estimate the number of women who start out with breast cancer at age 40, in advance of performing any mammographies.

Q: A Bayesian uses Bayes's Theorem to  update beliefs based on the evidence.  This requires that, even in advance of seeing the evidence, you have beliefs about what the evidence means - how likely you are to see the evidence, if various hypotheses are true - and how likely those hypotheses were, in advance of seeing the evidence.  To calculate a posterior probability using Bayes's Theorem, you need a prior probability and likelihood distribution.
Suppose you had a barrel containing some number of red and white balls.  If you start with the belief that each ball was independently assigned red color (vs. white color) at some fixed probability between 0 and 1, and you start out ignorant of this fixed probability (the parameter could anywhere between 0 and 1), then each red ball you see makes it more likely that the next ball will be red.  (By Laplace's Rule of Succession.)
On the other hand, if you start out with the prior belief that the barrel contains exactly 10 red balls and 10 white balls, then each red ball you see makes it less likely that the next ball will be red (because there are fewer red balls remaining).
Thus our prior can affect how we interpret the evidence.  The first prior is an inductive prior; things that happened before are predicted to happen again with greater probability.  The second prior is anti-inductive; the more red balls we see, the fewer we expect to see in the future.
In both cases, you started out believing something about the barrel - presumably because someone else told you, or because you saw it with your own eyes.  But then their words, or even your own eyesight, was evidence, and you must have had prior beliefs about probabilities and likelihoods in order to interpret the evidence.  So it seems that an ideal Bayesian would need some sort of inductive prior at the very moment they were born; and where an ideal Bayesian would get this prior, has occasionally been a matter of considerable controversy in the philosophy of probability.
The phrase  "Prior probability" (as opposed to "[...]") usually refers to a point estimate already based on considerable evidence - for example when we estimate the number of women who start out with breast cancer at age 40, in advance of performing any mammographies.
A: Priors

Q: Prisoner\'s dilemma
A: The Prisoner's dilemma is a classic problem in game theory. Two players independently have the option of cooperating with or defecting against the other player. If both players cooperate, they each receive a payoff C from a third party (a "banker" or benevolent nature); if they each defect, they both receive payoff D; if one cooperates and one defects, the defector receives payoff T and the cooperator receives payoff S. The prisoner's dilemma is the situation in which T > C > D > S. The payoffs are assumed to be in utils, each player only wants to maximize her own payoff, without any regard in either direction for the other player.
Notice that if you treat the other player's decision as completely independent from yours, if the other player defects, then you score higher if you defect as well, whereas if the other player cooperates, you do better by defecting. So it would seem that the rational decision would be to defect (at least if the game is to be played only once), and indeed, this is what classical causal decision theory says. And yet—and yet, if only somehow both players could agree to cooperate, they would both do better than if they both defected. If the players are timeless decision agents, they can.
Contents
1 Blog posts
2 External links
3 See also
4 References
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Blog posts
The True Prisoner's Dilemma
The Truly Iterated Prisoner's Dilemma
Re-formalizing PD by cousin_it
Blackmail, Nukes, and the Prisoner's Dilemma by Stuart Armstrong
Prisoner's Dilemma Tournament Results by prase
The continued misuse of the Prisoner's Dilemma by Silas Barta
Solution in ADT:

Q: The [...] is a classic problem in game theory. Two players independently have the option of cooperating with or defecting against the other player. If both players cooperate, they each receive a payoff C from a third party (a "banker" or benevolent nature); if they each defect, they both receive payoff D; if one cooperates and one defects, the defector receives payoff T and the cooperator receives payoff S. The [...] is the situation in which T > C > D > S. The payoffs are assumed to be in utils, each player only wants to maximize her own payoff, without any regard in either direction for the other player.
Notice that if you treat the other player's decision as completely independent from yours, if the other player defects, then you score higher if you defect as well, whereas if the other player cooperates, you do better by defecting. So it would seem that the rational decision would be to defect (at least if the game is to be played only once), and indeed, this is what classical causal decision theory says. And yet—and yet, if only somehow both players could agree to cooperate, they would both do better than if they both defected. If the players are timeless decision agents, they can.
Contents
1 Blog posts
2 External links
3 See also
4 References
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Blog posts
The True Prisoner's Dilemma
The Truly Iterated Prisoner's Dilemma
Re-formalizing PD by cousin_it
Blackmail, Nukes, and the Prisoner's Dilemma by Stuart Armstrong
Prisoner's Dilemma Tournament Results by prase
The continued misuse of the Prisoner's Dilemma by Silas Barta
Solution in ADT:
A: Prisoner\'s dilemma

Q: Privileging the hypothesis
A: The fallacy of  singling out a particular hypothesis for attention when there is insufficient evidence already in hand to justify such special attention.
To see the problem of privileging the hypothesis, suppose that the police in Largeville, a town with a million inhabitants, are investigating a murder in which there are few or no clues - the victim was stabbed to death in an alley, and there are no fingerprints and no witnesses.
Then, one of the detectives says, "Well... we have no idea who did it... no particular evidence singling out any of the million people in this city... but let's consider the possibility that this murder was committed by Mortimer Q. Snodgrass, who lives at 128 Ordinary Ln."
If the detective does not have evidence already in hand to justify singling out Mortimer for such special and individual attention, then this is, or ought to be, a violation of Mortimer's civil rights.
This is true even if the detective is not claiming that Mortimer did do it, but only suggesting that all the police officers consider that Mortimer might have done it.  The principle of locating the hypothesis says that it often takes more evidence to first distinguish the true hypothesis as worthy of explicit consideration, than to distinguish among the remaining alternatives.  So the detective is jumping over the job of providing most of the evidence that would have to be brought against Mortimer.
"Privileging the hypothesis" is the fallacy committed by a creationist who points out a purported flaw in standard evolutionary theory, and brings forward the Trinity to fill the gap - rather than a billion other deities or a trillion other naturalistic hypotheses.  Actually, without evidence already in hand that points to the Trinity specifically, one cannot justify raising that particular hypothesis to explicit attention rather than a trillion others.
The anti-work of anti-epistemology is to manufacture belief without evidence, and in large answer spaces, attention without evidence is more than halfway to belief without evidence.  Someone who spends all day pondering whether the Trinity does or does not exist - rather than Allah, Thor, or the Flying Spaghetti Monster - is more than halfway converted to Christianity.

Q: The fallacy of  singling out a particular hypothesis for attention when there is insufficient evidence already in hand to justify such special attention.
To see the problem of [...], suppose that the police in Largeville, a town with a million inhabitants, are investigating a murder in which there are few or no clues - the victim was stabbed to death in an alley, and there are no fingerprints and no witnesses.
Then, one of the detectives says, "Well... we have no idea who did it... no particular evidence singling out any of the million people in this city... but let's consider the possibility that this murder was committed by Mortimer Q. Snodgrass, who lives at 128 Ordinary Ln."
If the detective does not have evidence already in hand to justify singling out Mortimer for such special and individual attention, then this is, or ought to be, a violation of Mortimer's civil rights.
This is true even if the detective is not claiming that Mortimer did do it, but only suggesting that all the police officers consider that Mortimer might have done it.  The principle of locating the hypothesis says that it often takes more evidence to first distinguish the true hypothesis as worthy of explicit consideration, than to distinguish among the remaining alternatives.  So the detective is jumping over the job of providing most of the evidence that would have to be brought against Mortimer.
"[...]" is the fallacy committed by a creationist who points out a purported flaw in standard evolutionary theory, and brings forward the Trinity to fill the gap - rather than a billion other deities or a trillion other naturalistic hypotheses.  Actually, without evidence already in hand that points to the Trinity specifically, one cannot justify raising that particular hypothesis to explicit attention rather than a trillion others.
The anti-work of anti-epistemology is to manufacture belief without evidence, and in large answer spaces, attention without evidence is more than halfway to belief without evidence.  Someone who spends all day pondering whether the Trinity does or does not exist - rather than Allah, Thor, or the Flying Spaghetti Monster - is more than halfway converted to Christianity.
A: Privileging the hypothesis

Q: Problem of verifying rationality
A: The single largest problem for those desiring to create methods of systematically training for increased epistemic and instrumental rationality - how to verify that the training actually worked.  Very Awful Things happen to people who set out to build training methods and schools without strong means of verification - two hideous historical examples being modern-day practice of martial arts (once the teachers are no longer fighting duels to the death) and the proliferating "schools" of psychotherapy (in the entire absence of any experimental evidence that one school worked better than another, and indeed, in the presence of experimental evidence that the schools did not have any discernible difference in effectiveness).

Q: The single largest problem for those desiring to create methods of systematically training for increased epistemic and instrumental rationality - how to verify that the training actually worked.  Very Awful Things happen to people who set out to build training methods and schools without strong means of verification - two hideous historical examples being modern-day practice of martial arts (once the teachers are no longer fighting duels to the death) and the proliferating "schools" of psychotherapy (in the entire absence of any experimental evidence that one school worked better than another, and indeed, in the presence of experimental evidence that the schools did not have any discernible difference in effectiveness).
A: Problem of verifying rationality

Q: Prospect theory
A: An attempt by Amos Tversky and Daniel Kahneman to construct a decision theory that describes the actual behavior of humans, rather than idealized rational decision-making.
An agent, in prospect theory:

Q: An attempt by Amos Tversky and Daniel Kahneman to construct a decision theory that describes the actual behavior of humans, rather than idealized rational decision-making.
An agent, in [...]:
A: Prospect theory

Q: Quantum mechanics
A: The bad news: our ordinary world is made out of weird, fuzzy, unpredictable stuff. The good news: the weird, fuzzy, unpredictable stuff is made out of unfamiliar but perfectly sensible math.
People here have generally favored Hugh Everett's many-worlds interpretation over others.

Q: The bad news: our ordinary world is made out of weird, fuzzy, unpredictable stuff. The good news: the weird, fuzzy, unpredictable stuff is made out of unfamiliar but perfectly sensible math.
People here have generally favored Hugh Everett's many-worlds interpretation over others.
A: Quantum mechanics

Q: Radical honesty
A: Promoting less than maximally accurate beliefs is an act of sabotage. Don’t do it to anyone unless you’d also slash their tires, because they’re Nazis or whatever.
—Black Belt Bayesian
Radical honesty is a communication technique proposed by Brad Blanton in which discussion partners are not permitted to lie or deceive at all. Rather than being designed to enhance group epistemic rationality, radical honesty is designed to reduce stress and remove the layers of deceit that burden much of discourse.

Q: Promoting less than maximally accurate beliefs is an act of sabotage. Don’t do it to anyone unless you’d also slash their tires, because they’re Nazis or whatever.
—Black Belt Bayesian
[...] is a communication technique proposed by Brad Blanton in which discussion partners are not permitted to lie or deceive at all. Rather than being designed to enhance group epistemic rationality, [...] is designed to reduce stress and remove the layers of deceit that burden much of discourse.
A: Radical honesty

Q: Rational evidence
A: Rational evidence is the broadest possible sense of evidence, the Bayesian sense.  Rational evidence about a hypothesis H is any observation which has a different likelihood depending on whether H holds in reality or not.
Rational evidence is distinguished from narrower forms of evidence, such as scientific evidence or legal evidence.  For a belief to be scientific, you should be able to do repeatable experiments to verify the belief.  For evidence to be admissible in court, it must e.g. be a personal observation rather than hearsay.
For example, suppose I tell you that the original author of this paragraph wore white socks while writing it.  (In fact, I do so tell you.)  You now have rational evidence that the author of this paragraph wore white socks.  But it is not scientific knowledge because there is no experiment you can do for yourself to verify whether it is true.  And it is not legal evidence - you could testify in court that I had told you my socks were white, but you could not testify that my socks were white.
The scientific method can be viewed as a special standard of admissible evidence protecting a pool of extra-strong beliefs.  Conversely, a fact can be rationally guessable without it generating the specially strong evidence which would qualify knowledge of the fact as "scientific".  Just as a police detective may rationally know the identity of the local crime boss without having the special evidence needed to prove it in court.

Q: [...] is the broadest possible sense of evidence, the Bayesian sense.  [...] about a hypothesis H is any observation which has a different likelihood depending on whether H holds in reality or not.
[...] is distinguished from narrower forms of evidence, such as scientific evidence or legal evidence.  For a belief to be scientific, you should be able to do repeatable experiments to verify the belief.  For evidence to be admissible in court, it must e.g. be a personal observation rather than hearsay.
For example, suppose I tell you that the original author of this paragraph wore white socks while writing it.  (In fact, I do so tell you.)  You now have [...] that the author of this paragraph wore white socks.  But it is not scientific knowledge because there is no experiment you can do for yourself to verify whether it is true.  And it is not legal evidence - you could testify in court that I had told you my socks were white, but you could not testify that my socks were white.
The scientific method can be viewed as a special standard of admissible evidence protecting a pool of extra-strong beliefs.  Conversely, a fact can be rationally guessable without it generating the specially strong evidence which would qualify knowledge of the fact as "scientific".  Just as a police detective may rationally know the identity of the local crime boss without having the special evidence needed to prove it in court.
A: Rational evidence

Q: Rationalist taboo
A: Rationalist taboo is a technique for fighting muddles in discussions. By prohibiting the use of a certain word and all the words synonymous to it, people are forced to elucidate the specific contextual meaning they want to express, thus removing ambiguity otherwise present in a single word.

Q: [...] is a technique for fighting muddles in discussions. By prohibiting the use of a certain word and all the words synonymous to it, people are forced to elucidate the specific contextual meaning they want to express, thus removing ambiguity otherwise present in a single word.
A: Rationalist taboo

Q: Rationality is systematized winning
A: You may try to name the highest principle with names such as “the map that reflects the territory” or “experience of success and failure” or “Bayesian decision theory”. But perhaps you describe incorrectly the nameless virtue. How will you discover your mistake? Not by comparing your description to itself, but by comparing it to that which you did not name.
—Twelve Virtues of Rationality
The point of all this discussion of rationality is to actually achieve truer beliefs and more effective actions. It's not some arbitrary social fashion; there are actual criteria of success. It is for this reason that it is written that rationalists should win. If some particular ritual of cognition—even one that you have long cherished as "rational"—systematically gives poorer results relative to some alternative, it is not rational to cling to it. The rational algorithm is to do what works, to get the actual answer—in short, to win, whatever the method, whatever the means. If you can detect a systematic mistake in your thinking, then fix it; if you can see a better method, then adopt it.

Q: You may try to name the highest principle with names such as “the map that reflects the territory” or “experience of success and failure” or “Bayesian decision theory”. But perhaps you describe incorrectly the nameless virtue. How will you discover your mistake? Not by comparing your description to itself, but by comparing it to that which you did not name.
—Twelve Virtues of Rationality
The point of all this discussion of rationality is to actually achieve truer beliefs and more effective actions. It's not some arbitrary social fashion; there are actual criteria of success. It is for this reason that it is written that rationalists should win. If some particular ritual of cognition—even one that you have long cherished as "rational"—systematically gives poorer results relative to some alternative, it is not rational to cling to it. The rational algorithm is to do what works, to get the actual answer—in short, to win, whatever the method, whatever the means. If you can detect a systematic mistake in your thinking, then fix it; if you can see a better method, then adopt it.
A: Rationality is systematized winning

Q: Rationality and Philosophy
A: A sequence by lukeprog examining the implications of rationality and cognitive science for philosophical method.

Q: A sequence by lukeprog examining the implications of rationality and cognitive science for philosophical method.
A: Rationality and Philosophy

Q: Rationality as martial art
A: A metaphor for rationality as the martial art of mind; training brains in the same fashion as muscles.
The metaphor is intended to have complex connotations, rather than being strictly positive.  Do modern-day martial arts suffer from being insufficiently tested in realistic fighting, and do attempts at rationality training run into the same problem?

Q: A metaphor for rationality as the martial art of mind; training brains in the same fashion as muscles.
The metaphor is intended to have complex connotations, rather than being strictly positive.  Do modern-day martial arts suffer from being insufficiently tested in realistic fighting, and do attempts at rationality training run into the same problem?
A: Rationality as martial art

Q: Rationality materials
A: If you're new to Less Wrong, a good place to start is with the Sequences. The Core Sequences are the ones that focus on the fundamentals of rationality.
The Sequence index lists sequence posts in logical order. There is also a dependency tree of all of Eliezer Yudkowsky's posts from November 2006 to December 2008, and a couple of "abridged" versions of the Sequences: 1, 2.
Here is a sampling of various sequence posts:
Core Sequences
Map and Territory
What Do We Mean By "Rationality"?
What is Evidence?
An Intuitive Explanation of Bayes' Theorem (at yudkowsky.net)
Mysterious Answers to Mysterious Questions
Making Beliefs Pay Rent
Applause lights
Hindsight Devalues Science
Mysterious Answers to Mysterious Questions
A Human's Guide to Words
Taboo Your Words
Disguised Queries
How An Algorithm Feels From Inside
How To Actually Change Your Mind
A Fable of Science and Politics
The Fallacy of Gray
Politics is the Mind-Killer
Knowing About Biases Can Hurt People
The Bottom Line
The Proper Use of Humility
No One Can Exempt You From Rationality's Laws
Reductionism
Mind Projection Fallacy
Righting a Wrong Question
Explaining vs. Explaining Away
Reductionism
Other Sequences
Positivism, Self Deception, and Neuroscience
The Apologist and the Revolutionary
How Not to Lose an Argument
The Mystery of the Haunted Rationalist
Decision Theory of Newcomblike Problems
Confusion About Newcomb is Confusion About Counterfactuals
The Science of Winning at Life
Scientific Self-Help: The State of Our Knowledge
How to Beat Procrastination
How to Be Happy
Metaethics sequence
Where Recursive Justification Hits Bottom
No Universally Compelling Arguments
Sorting Pebbles Into Correct Heaps
Priming and Implicit Association
Never Leave Your Room
Fun theory
Eutopia is Scary
Complex Novelty
Living Luminously
Seven Shiny Stories
Ureshiku Naritai
You Are Likely To Be Eaten By A Grue
Challenging the Difficult
Tsuyoku Naritai! (I Want To Become Stronger)
Something to Protect
Decision Analysis
Value of Information: Four Examples
5 Axioms of Decision Making
Rationality and Philosophy
Philosophy: A Diseased Discipline
Philosophy by Humans, 1
The Craft and the Community
Why Our Kind Can't Cooperate
Raising the Sanity Waterline
Quantum Physics
Quantum Explanations
Can You Prove Two Particles Are Identical?
Here are some "standalone" posts that are worth reading:
Heuristics and biases
The Cognitive Science of Rationality
Conjunction Fallacy
Anchoring and Adjustment
The Bias You Didn't Expect
Epistemology
Epistemic Luck
The Power of Positivist Thinking
Identity and signalling
A Theory of Identity (at overcomingbias.com)
Cached Selves
The Curse of Identity
Have no Heroes, and no Villains
Don't Revere The Bearer of Good Info
Intellectual Hipsters and Meta-Contrarianism
Keep Your Identity Small (at paulgraham.com)
Scholarship and learning
Understanding Your Understanding
The Neglected Virtue of Scholarship
Some Heuristics for Evaluating the Soundness of the Academic Mainstream in Unfamiliar Fields
Moral philosophy
Torture vs. Dust Specks
A Much Better Life?
Doing your good deed for the day
Statistics
What is Bayesianism?
Simpson's Paradox
Frequentist Statistics are Frequently Subjective
Charity
Money: The Unit of Caring
Efficient Charity: Do Unto Others...
Optimizing Fuzzies And Utilons: The Altruism Chip Jar
Belief calibration
Confidence levels inside and outside an argument
Self-fulfilling correlations
1001 PredictionBook Nights
Privileging the hypothesis
Decision theory
Decision Theories: A Less Wrong Primer
Newcomb's Problem and Regret of Rationality
Eight Short Studies on Excuses
If a tree falls on Sleeping Beauty...
Akrasia, motivation, and self-improvement
Ugh fields
Humans are not automatically strategic
Levels of Action
Action and habit
Compartmentalization in epistemic and instrumental rationality
A Suite of Pragmatic Considerations in Favor of Niceness
Existential risk and risks from artificial intelligence
So You Want To Save The World
That Alien Message
The Hanson-Yudkowsky AI-Foom Debate
If you're interested in starting a Less Wrong meetup, How to Run a Successful Less Wrong Meetup is a guide for meetup organizers (pdf here).

Q: If you're new to Less Wrong, a good place to start is with the Sequences. The Core Sequences are the ones that focus on the fundamentals of rationality.
The Sequence index lists sequence posts in logical order. There is also a dependency tree of all of Eliezer Yudkowsky's posts from November 2006 to December 2008, and a couple of "abridged" versions of the Sequences: 1, 2.
Here is a sampling of various sequence posts:
Core Sequences
Map and Territory
What Do We Mean By "Rationality"?
What is Evidence?
An Intuitive Explanation of Bayes' Theorem (at yudkowsky.net)
Mysterious Answers to Mysterious Questions
Making Beliefs Pay Rent
Applause lights
Hindsight Devalues Science
Mysterious Answers to Mysterious Questions
A Human's Guide to Words
Taboo Your Words
Disguised Queries
How An Algorithm Feels From Inside
How To Actually Change Your Mind
A Fable of Science and Politics
The Fallacy of Gray
Politics is the Mind-Killer
Knowing About Biases Can Hurt People
The Bottom Line
The Proper Use of Humility
No One Can Exempt You From Rationality's Laws
Reductionism
Mind Projection Fallacy
Righting a Wrong Question
Explaining vs. Explaining Away
Reductionism
Other Sequences
Positivism, Self Deception, and Neuroscience
The Apologist and the Revolutionary
How Not to Lose an Argument
The Mystery of the Haunted Rationalist
Decision Theory of Newcomblike Problems
Confusion About Newcomb is Confusion About Counterfactuals
The Science of Winning at Life
Scientific Self-Help: The State of Our Knowledge
How to Beat Procrastination
How to Be Happy
Metaethics sequence
Where Recursive Justification Hits Bottom
No Universally Compelling Arguments
Sorting Pebbles Into Correct Heaps
Priming and Implicit Association
Never Leave Your Room
Fun theory
Eutopia is Scary
Complex Novelty
Living Luminously
Seven Shiny Stories
Ureshiku Naritai
You Are Likely To Be Eaten By A Grue
Challenging the Difficult
Tsuyoku Naritai! (I Want To Become Stronger)
Something to Protect
Decision Analysis
Value of Information: Four Examples
5 Axioms of Decision Making
Rationality and Philosophy
Philosophy: A Diseased Discipline
Philosophy by Humans, 1
The Craft and the Community
Why Our Kind Can't Cooperate
Raising the Sanity Waterline
Quantum Physics
Quantum Explanations
Can You Prove Two Particles Are Identical?
Here are some "standalone" posts that are worth reading:
Heuristics and biases
The Cognitive Science of Rationality
Conjunction Fallacy
Anchoring and Adjustment
The Bias You Didn't Expect
Epistemology
Epistemic Luck
The Power of Positivist Thinking
Identity and signalling
A Theory of Identity (at overcomingbias.com)
Cached Selves
The Curse of Identity
Have no Heroes, and no Villains
Don't Revere The Bearer of Good Info
Intellectual Hipsters and Meta-Contrarianism
Keep Your Identity Small (at paulgraham.com)
Scholarship and learning
Understanding Your Understanding
The Neglected Virtue of Scholarship
Some Heuristics for Evaluating the Soundness of the Academic Mainstream in Unfamiliar Fields
Moral philosophy
Torture vs. Dust Specks
A Much Better Life?
Doing your good deed for the day
Statistics
What is Bayesianism?
Simpson's Paradox
Frequentist Statistics are Frequently Subjective
Charity
Money: The Unit of Caring
Efficient Charity: Do Unto Others...
Optimizing Fuzzies And Utilons: The Altruism Chip Jar
Belief calibration
Confidence levels inside and outside an argument
Self-fulfilling correlations
1001 PredictionBook Nights
Privileging the hypothesis
Decision theory
Decision Theories: A Less Wrong Primer
Newcomb's Problem and Regret of Rationality
Eight Short Studies on Excuses
If a tree falls on Sleeping Beauty...
Akrasia, motivation, and self-improvement
Ugh fields
Humans are not automatically strategic
Levels of Action
Action and habit
Compartmentalization in epistemic and instrumental rationality
A Suite of Pragmatic Considerations in Favor of Niceness
Existential risk and risks from artificial intelligence
So You Want To Save The World
That Alien Message
The Hanson-Yudkowsky AI-Foom Debate
If you're interested in starting a Less Wrong meetup, How to Run a Successful Less Wrong Meetup is a guide for meetup organizers (pdf here).
A: Rationality materials

Q: Rationality power tools
A: A Rationality power tool is a computer program that assists with rationality, either epistemic or instrumental. They are not games - power tools are tools, and have the flexibility to be applied to multiple situations while guiding or enhancing their users' power to win or "improve their understanding."
Contents
1 Existing tools
1.1 Coordinated efforts
1.2 Debate tools
1.3 Greasemonkey scripts and proxies
1.4 Prediction tracking
1.4.1 Prediction markets
1.5 Mathematical
1.6 Time management
1.6.1 Document savers
1.7 Spaced repetition
1.8 Personal data tallying/Self experimentation
1.9 Other
2 Defunct
3 In development
4 Proposed
5 Blog post
6 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
  Existing tools
  Coordinated efforts
BitCoin – Cryptocurrency
ChipIn – For raising funding for projects.
ThePoint – Non-financial coordination.
KickStarter – For raising funding for projects. Donations are collected only if a certain threshold is reached.
  Debate tools
Main article: Debate tools
TakeOnIt – A database of expert opinion.
  Greasemonkey scripts and proxies
These programs change the behavior or appearance of websites.
LessWrong Power Reader
  Prediction tracking
PredictionBook – "a site to record and track your predictions."
Prediction markets
Foresight Exchange – Free website; a funny-money prediction market, ancestor of Consensus Point and The Idea Futures package
Intrade – Real money prediction market
ipredict – New Zealand based prediction market.
Zocalo – software to set up a prediction market
  Mathematical
Gambit – "Gambit is a set of software tools for doing computation on finite, noncooperative games."
  Time management
Org-mode – for Emacs. "org-mode can do clocking (to see how long I'm spending on tasks), all kinds of reminders, todo-lists and can even check which of my 'projects' currently don't have a NextAction defined" - djcb
  Document savers
Discussed in "Avoid interruptions by time-shifting them"
Instapaper
Read it Later
  Spaced repetition
Spaced repetition is a method for memorizing important material.

Q: A Rationality power tool is a computer program that assists with rationality, either epistemic or instrumental. They are not games - power tools are tools, and have the flexibility to be applied to multiple situations while guiding or enhancing their users' power to win or "improve their understanding."
Contents
1 Existing tools
1.1 Coordinated efforts
1.2 Debate tools
1.3 Greasemonkey scripts and proxies
1.4 Prediction tracking
1.4.1 Prediction markets
1.5 Mathematical
1.6 Time management
1.6.1 Document savers
1.7 Spaced repetition
1.8 Personal data tallying/Self experimentation
1.9 Other
2 Defunct
3 In development
4 Proposed
5 Blog post
6 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
  Existing tools
  Coordinated efforts
BitCoin – Cryptocurrency
ChipIn – For raising funding for projects.
ThePoint – Non-financial coordination.
KickStarter – For raising funding for projects. Donations are collected only if a certain threshold is reached.
  Debate tools
Main article: Debate tools
TakeOnIt – A database of expert opinion.
  Greasemonkey scripts and proxies
These programs change the behavior or appearance of websites.
LessWrong Power Reader
  Prediction tracking
PredictionBook – "a site to record and track your predictions."
Prediction markets
Foresight Exchange – Free website; a funny-money prediction market, ancestor of Consensus Point and The Idea Futures package
Intrade – Real money prediction market
ipredict – New Zealand based prediction market.
Zocalo – software to set up a prediction market
  Mathematical
Gambit – "Gambit is a set of software tools for doing computation on finite, noncooperative games."
  Time management
Org-mode – for Emacs. "org-mode can do clocking (to see how long I'm spending on tasks), all kinds of reminders, todo-lists and can even check which of my 'projects' currently don't have a NextAction defined" - djcb
  Document savers
Discussed in "Avoid interruptions by time-shifting them"
Instapaper
Read it Later
  Spaced repetition
Spaced repetition is a method for memorizing important material.
A: Rationality power tools

Q: Rationalization
A: A curious term for the very opposite and antithesis of rationality, as if lying were called "truthization".  You cannot "rationalize" what is not already rational.
Rationality starts from evidence, and then crunches forward through belief updates, in order to output a probable conclusion.  "Rationalization" starts from a conclusion, and then works backward to arrive at arguments apparently favoring that conclusion.  Rationalization argues for a side already selected; rationality tries to choose between sides.
Rationalization can be conscious or unconscious.  It can take on a blatant, conscious form, in which you are aware that you want a particular side to be correct and you deliberately compose arguments for only that side, looking over the evidence and consciously filtering which facts will be presented.  Or it can occur at perceptual speeds, without conscious intent or conscious awareness.
Defeating rationalization - or even discovering rationalizations - is a lifelong battle for the aspiring rationalist.

Q: A curious term for the very opposite and antithesis of rationality, as if lying were called "truthization".  You cannot "rationalize" what is not already rational.
Rationality starts from evidence, and then crunches forward through belief updates, in order to output a probable conclusion.  "[...]" starts from a conclusion, and then works backward to arrive at arguments apparently favoring that conclusion.  [...] argues for a side already selected; rationality tries to choose between sides.
[...] can be conscious or unconscious.  It can take on a blatant, conscious form, in which you are aware that you want a particular side to be correct and you deliberately compose arguments for only that side, looking over the evidence and consciously filtering which facts will be presented.  Or it can occur at perceptual speeds, without conscious intent or conscious awareness.
Defeating [...] - or even discovering [...]s - is a lifelong battle for the aspiring rationalist.
A: Rationalization

Q: Reality is normal
A: There is a shattering truth, so surprising and terrifying that people resist the implications with all their strength. Yet there are a lonely few with the courage to accept this satori. Here is wisdom, if you would be wise:
Since the beginning
Not one unusual thing
Has ever happened.
—Eliezer Yudkowsky, A Technical Explanation of Technical Explanation
If at any point some truth should seem shocking or bizarre, it's your brain that labels it as such; reality simply is what it is. Surprise is the measure of a poor hypothesis; a good model puts probability mass on outcomes that are more likely to actually occur. If your model of reality consistently makes bad predictions — if the the real world seems utterly strange or surreal to your intuitions — then it's your intuitions that need to change; the world isn't going to. Reality is normal.

Q: There is a shattering truth, so surprising and terrifying that people resist the implications with all their strength. Yet there are a lonely few with the courage to accept this satori. Here is wisdom, if you would be wise:
Since the beginning
Not one unusual thing
Has ever happened.
—Eliezer Yudkowsky, A Technical Explanation of Technical Explanation
If at any point some truth should seem shocking or bizarre, it's your brain that labels it as such; reality simply is what it is. Surprise is the measure of a poor hypothesis; a good model puts probability mass on outcomes that are more likely to actually occur. If your model of reality consistently makes bad predictions — if the the real world seems utterly strange or surreal to your intuitions — then it's your intuitions that need to change; the world isn't going to. [...].
A: Reality is normal

Q: Reductionism
A: Reductionism is a disbelief that the higher levels of simplified multilevel models are out there in the territory, that concepts constructed by mind in themselves play a role in the behavior of reality. This doesn't contradict the notion that the concepts used in simplified multilevel models refer to the actual clusters of configurations of reality.

Q: [...] is a disbelief that the higher levels of simplified multilevel models are out there in the territory, that concepts constructed by mind in themselves play a role in the behavior of reality. This doesn't contradict the notion that the concepts used in simplified multilevel models refer to the actual clusters of configurations of reality.
A: Reductionism

Q: Reductionism (sequence)
A: How to take reality apart into pieces... and live in that universe, where we have always lived, without feeling disappointed about the fact that complicated things are made of simpler things.
Definitely one of the core LW sequences. Written by Eliezer Yudkowsky, ran from 08 Mar 2008 to 05 Apr 2008, with scattered other posts elsewhere.
Items in italic are potentially skippable; items in bold are extra-important.
This sequence follows Mysterious Answers to Mysterious Questions and requires the following posts from it:
Making Beliefs Pay Rent
The Virtue of Narrowness
Hindsight Bias
Fake Explanations
Guessing the Teacher's Password
Fake Causality
Mysterious Answers to Mysterious Questions
The Futility of Emergence
Explain/Worship/Ignore
This sequence follows A Human's Guide to Words and requires at least the following posts from it:
How An Algorithm Feels From Inside
Feel the Meaning
Taboo Your Words
Replace the Symbol with the Substance
Early posts out of sequence:

Q: How to take reality apart into pieces... and live in that universe, where we have always lived, without feeling disappointed about the fact that complicated things are made of simpler things.
Definitely one of the core LW sequences. Written by Eliezer Yudkowsky, ran from 08 Mar 2008 to 05 Apr 2008, with scattered other posts elsewhere.
Items in italic are potentially skippable; items in bold are extra-important.
This sequence follows Mysterious Answers to Mysterious Questions and requires the following posts from it:
Making Beliefs Pay Rent
The Virtue of Narrowness
Hindsight Bias
Fake Explanations
Guessing the Teacher's Password
Fake Causality
Mysterious Answers to Mysterious Questions
The Futility of Emergence
Explain/Worship/Ignore
This sequence follows A Human's Guide to Words and requires at least the following posts from it:
How An Algorithm Feels From Inside
Feel the Meaning
Taboo Your Words
Replace the Symbol with the Substance
Early posts out of sequence:
A: Reductionism (sequence)

Q: Reflective decision theory
A: Reflective decision theory is a term occasionally used to refer to a decision theory which does not cause an agent to regret having used it. Such a regret would be a Reflective inconsistency, as seen in the Causal Decision Theorist who regrets not being able to achieve the optimal result in Newcomb's problem.
Many hypothesized AGIs are expected to be powerful specifically due to an ability to access their own source code and self-modify. Because such an AGI could change its decision algorithm in a situation like Newcomb's Problem, it is necessary to develop a reflectively consistent decision theory to understand the AGI's behavior. Particularly, reflective consistency would be needed to ensure that an AGI preserved a Friendly value system throughout its self-modifications.
For the reasons above, this is a topic of interest to SIAI's research team. Proposed solutions include Eliezer Yudkowsky's Timeless Decision Theory.

Q: [...] is a term occasionally used to refer to a decision theory which does not cause an agent to regret having used it. Such a regret would be a Reflective inconsistency, as seen in the Causal Decision Theorist who regrets not being able to achieve the optimal result in Newcomb's problem.
Many hypothesized AGIs are expected to be powerful specifically due to an ability to access their own source code and self-modify. Because such an AGI could change its decision algorithm in a situation like Newcomb's Problem, it is necessary to develop a reflectively consistent decision theory to understand the AGI's behavior. Particularly, reflective consistency would be needed to ensure that an AGI preserved a Friendly value system throughout its self-modifications.
For the reasons above, this is a topic of interest to SIAI's research team. Proposed solutions include Eliezer Yudkowsky's Timeless Decision Theory.
A: Reflective decision theory

Q: Religion
A: This article is designed to be a soft primer for the community view of religion, atheism, theism, spirituality, and sacredness.
Religion is a complex group of human activities — involving tribal affiliation, belief in belief, supernatural claims, and a range of shared group practices such as worship meetings, rites of passage, etc.

Q: This article is designed to be a soft primer for the community view of [...], atheism, theism, spirituality, and sacredness.
[...] is a complex group of human activities — involving tribal affiliation, belief in belief, supernatural claims, and a range of shared group practices such as worship meetings, rites of passage, etc.
A: Religion

Q: Representativeness heuristic
A: The Representativeness Heuristic is a rule of thumb wherein people judge the probability or frequency of a hypothesis by considering how much the hypothesis resembles available data as opposed to using a Bayesian calculation. While often very useful in everyday life, it can also result in neglect of relevant base rates and other cognitive biases. The representative heuristic was first proposed by Amos Tversky and Daniel Kahneman.

Q: The Representativeness Heuristic is a rule of thumb wherein people judge the probability or frequency of a hypothesis by considering how much the hypothesis resembles available data as opposed to using a Bayesian calculation. While often very useful in everyday life, it can also result in neglect of relevant base rates and other cognitive biases. The representative heuristic was first proposed by Amos Tversky and Daniel Kahneman.
A: Representativeness heuristic

Q: Reversed stupidity is not intelligence
A: The world's greatest fool may say the Sun is shining, but that doesn't make it dark out.
—Robert Pirsig
It takes a lot of evidence and rationality just to locate a good hypothesis in the search space. Getting it right is a precise Art, whereas there are a million ways to get it wrong. A stopped clock is right twice a day; answering true-or-false questions at entirely at random will still yield a fifty percent success rate. To do better, you can't just perform a reversal of an obvious fallacy.  Reversed stupidity is not intelligence; faulty reasoning does not produce "antitrue" conclusions that can just be negated; the true theory should make sense on its own terms; not just as a refutation of some nonsense.

Q: The world's greatest fool may say the Sun is shining, but that doesn't make it dark out.
—Robert Pirsig
It takes a lot of evidence and rationality just to locate a good hypothesis in the search space. Getting it right is a precise Art, whereas there are a million ways to get it wrong. A stopped clock is right twice a day; answering true-or-false questions at entirely at random will still yield a fifty percent success rate. To do better, you can't just perform a reversal of an obvious fallacy.  [...]; faulty reasoning does not produce "antitrue" conclusions that can just be negated; the true theory should make sense on its own terms; not just as a refutation of some nonsense.
A: Reversed stupidity is not intelligence

Q: Scales of justice fallacy
A: Lady Justice is widely depicted as carrying a scales. A scales has the property that whatever pulls one side down, pushes the other side up. This makes things very convenient and easy to track. It's also usually a gross distortion.
In human discourse there is a natural tendency to treat discussion as a form of combat, an extension of war, a sport; and in sports you only need to keep track of how many points have been scored by each team. There are only two sides, and every point scored against one side, is a point in favor of the other. Everyone in the audience keeps a mental running count of how many points each speaker scores against the other. At the end of the debate, the speaker who has scored more points is, obviously, the winner; so everything he says must be true, and everything the loser says must be wrong.
—The Scales of Justice, the Notebook of Rationality
The scales of justice fallacy refers to the error of using a simple polarized scheme for deciding a complex issue: each piece of evidence about the question is individually categorized as supporting exactly one of the two opposing positions. This scheme distorts the conclusions that can be drawn from each piece of evidence, doesn't allow to take into account the dependencies between the pieces of evidence in the context of the issue, and biases perception of individual pieces of evidence, making the ones that fall into the same category seem to support each other, and ones falling in the opposite categories to seem to oppose each other. In reality, individual pieces of evidence may be separate factual claims, arrived at independently, with the issue depending on specific combinations of facts.

Q: Lady Justice is widely depicted as carrying a scales. A scales has the property that whatever pulls one side down, pushes the other side up. This makes things very convenient and easy to track. It's also usually a gross distortion.
In human discourse there is a natural tendency to treat discussion as a form of combat, an extension of war, a sport; and in sports you only need to keep track of how many points have been scored by each team. There are only two sides, and every point scored against one side, is a point in favor of the other. Everyone in the audience keeps a mental running count of how many points each speaker scores against the other. At the end of the debate, the speaker who has scored more points is, obviously, the winner; so everything he says must be true, and everything the loser says must be wrong.
—The Scales of Justice, the Notebook of Rationality
The [...] refers to the error of using a simple polarized scheme for deciding a complex issue: each piece of evidence about the question is individually categorized as supporting exactly one of the two opposing positions. This scheme distorts the conclusions that can be drawn from each piece of evidence, doesn't allow to take into account the dependencies between the pieces of evidence in the context of the issue, and biases perception of individual pieces of evidence, making the ones that fall into the same category seem to support each other, and ones falling in the opposite categories to seem to oppose each other. In reality, individual pieces of evidence may be separate factual claims, arrived at independently, with the issue depending on specific combinations of facts.
A: Scales of justice fallacy

Q: Scholarship
A: The eleventh virtue is scholarship. Study many sciences and absorb their power as your own. Each field that you consume makes you larger. If you swallow enough sciences the gaps between them will diminish and your knowledge will become a unified whole. If you are gluttonous you will become vaster than mountains. It is especially important to eat math and science which impinges upon rationality: Evolutionary psychology, heuristics and biases, social psychology, probability theory, decision theory. But these cannot be the only fields you study. The Art must have a purpose other than itself, or it collapses into infinite recursion.

Q: The eleventh virtue is [...]. Study many sciences and absorb their power as your own. Each field that you consume makes you larger. If you swallow enough sciences the gaps between them will diminish and your knowledge will become a unified whole. If you are gluttonous you will become vaster than mountains. It is especially important to eat math and science which impinges upon rationality: Evolutionary psychology, heuristics and biases, social psychology, probability theory, decision theory. But these cannot be the only fields you study. The Art must have a purpose other than itself, or it collapses into infinite recursion.
A: Scholarship

Q: Science
A: Science is a method for developing true beliefs about the world. It works by developing hypotheses about the world, creating experiments that would allow the hypotheses to be tested, and running the experiments. By having people publish their falsifiable predictions and their experimental results, science protects itself from individuals deceiving themselves or others.

Q: [...] is a method for developing true beliefs about the world. It works by developing hypotheses about the world, creating experiments that would allow the hypotheses to be tested, and running the experiments. By having people publish their falsifiable predictions and their experimental results, [...] protects itself from individuals deceiving themselves or others.
A: Science

Q: Scope insensitivity
A: The human brain can't emotionally represent large quantities; an environmental measure that will save 200,000 birds doesn't conjure anywhere near a hundred times the emotional impact and willingness-to-pay of a measure that would save 2,000 birds, even though in fact the former measure is two orders of magnitude more effective.

Q: The human brain can't emotionally represent large quantities; an environmental measure that will save 200,000 birds doesn't conjure anywhere near a hundred times the emotional impact and willingness-to-pay of a measure that would save 2,000 birds, even though in fact the former measure is two orders of magnitude more effective.
A: Scope insensitivity

Q: Scoring rule
A: In decision theory, a scoring rule is a measure of someone's performance at making predictions under uncertainty (which comprises two distinct aspects, calibration and discrimination). A proper scoring rule encourages the forecaster to be honest, as expected payoff is maximized by accurately reporting personal belief about the predicted event, rather than by gaming the system.

Q: In decision theory, a [...] is a measure of someone's performance at making predictions under uncertainty (which comprises two distinct aspects, calibration and discrimination). A proper [...] encourages the forecaster to be honest, as expected payoff is maximized by accurately reporting personal belief about the predicted event, rather than by gaming the system.
A: Scoring rule

Q: Seeing with Fresh Eyes
A: A sequence on the incredibly difficult feat of getting your brain to actually think about something, instead of instantly stopping on the first thought that comes to mind.
This is sometimes referred to as "thinking outside the box" by people who, for your convenience, will go on to helpfully point out exactly where "outside the box" is located.  The Less Wrong version is called "thinking outside the 'Outside the Box' box".  Isn't it funny how nonconformists all dress the same...
Subsequence of How To Actually Change Your Mind.

Q: A sequence on the incredibly difficult feat of getting your brain to actually think about something, instead of instantly stopping on the first thought that comes to mind.
This is sometimes referred to as "thinking outside the box" by people who, for your convenience, will go on to helpfully point out exactly where "outside the box" is located.  The Less Wrong version is called "thinking outside the 'Outside the Box' box".  Isn't it funny how nonconformists all dress the same...
Subsequence of How To Actually Change Your Mind.
A: Seeing with Fresh Eyes

Q: Self-deception
A: Self-deception is a state of preserving a wrong belief, often facilitated by denying or rationalizing away the relevance, significance, or importance of opposing evidence and logical arguments. Beliefs supported by self-deception are often chosen for reasons other than how closely those beliefs approximate truth.
Deceiving yourself is harder than it seems. What looks like a successively adopted false belief may actually be just a belief in false belief.

Q: [...] is a state of preserving a wrong belief, often facilitated by denying or rationalizing away the relevance, significance, or importance of opposing evidence and logical arguments. Beliefs supported by [...] are often chosen for reasons other than how closely those beliefs approximate truth.
Deceiving yourself is harder than it seems. What looks like a successively adopted false belief may actually be just a belief in false belief.
A: Self-deception

Q: Sequences
A: A sequence is a series of multiple posts on Less Wrong on the same topic, to coherently and fully explore a particular thesis.
Reading the sequences is the most systematic way to approach the Less Wrong archives.
See also an alternative index of the sequences, and another for people who already have a science background.
If you prefer books over blog posts, Thinking and Deciding by Jonathan Baron and Good and Real by Gary Drescher have both been mentioned as books with lots of overlap with the sequences.
Contents
1 Alternative formats
2 Translations
3 Core Sequences
4 Major Sequences
4.1 Mysterious Answers to Mysterious Questions
4.2 A Human's Guide to Words
4.3 How To Actually Change Your Mind
4.4 Reductionism
4.5 The Quantum Physics Sequence
4.6 The Metaethics Sequence
4.7 The Fun Theory Sequence
4.8 The Craft and the Community
5 Minor Sequences
5.1 Map and Territory
5.2 Seeing with Fresh Eyes
5.3 Politics is the Mind-Killer
5.4 Death Spirals and the Cult Attractor
5.5 Ethical Injunctions
5.6 Joy in the Merely Real
5.7 Zombies Sequence
5.8 The Simple Math of Evolution
5.9 Challenging the Difficult
5.10 Coming of Age
6 Sequences by Others
6.1 Positivism, Self Deception, and Neuroscience by Yvain
6.2 Priming and Implicit Association by Yvain
6.3 Decision Theory of Newcomblike Problems by AnnaSalamon
6.4 Living Luminously by Alicorn
6.5 The Science of Winning at Life by lukeprog
6.6 Rationality and Philosophy by lukeprog
6.7 No-Nonsense Metaethics by lukeprog
6.8 What Intelligence Tests Miss by Kaj_Sotala
6.9 Why Everyone (Else) Is a Hypocrite by Kaj_Sotala
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Alternative formats
The Sequences have been converted to eReader compatible formats by several projects.
Print ready versions by jb55 (GitHub). Has versions in Markdown, PDF, and ePub. ePubs can be converted to nearly any other format with calibre.
lw2ebook by OneWhoFrogs (GitHub). Includes all sequences in ePub and mobi formats.
Sequences in EPub and MOBI formats by ciphergoth
Print ready versions by Jordan. Contains all posts of each sequence in one HTML file.
Translations
Main article: Translations into other languages
French
Italian
Spanish
Russian
Core Sequences
Map and Territory contains some of the most important introductory posts and essays.
Read Mysterious Answers to Mysterious Questions and  Reductionism in order to ensure you're not making any basic rationality errors.
The most important technique that Less Wrong can offer you is How To Actually Change Your Mind.
Major Sequences
Long sequences that have been completed and organized into a guide.
Mysterious Answers to Mysterious Questions
How to see through the many disguises of answers or beliefs or statements that remove curiosity without alleviating confusion.
A Human's Guide to Words
Sequence guide: 37 Ways That Words Can Be Wrong
List of posts: A Human's Guide to Words
A series on the use and abuse of words; why you can't define a word any way you like; how human brains seem to process definitions.  First introduces the Mind projection fallacy and the concept of how an algorithm feels from inside, which makes it a basic intro to key elements of the LW zeitgeist.
How To Actually Change Your Mind
A mega-sequence scattered over almost all of Less Wrong on the ultra-high-level penultimate technique of rationality: actually updating on evidence.
Organized into eight subsequences.
  Reductionism
How to take reality apart into pieces... and live in that universe, where we have always lived, without feeling disappointed about the fact that complicated things are made of simpler things.
The Quantum Physics Sequence
A non-mysterious introduction to quantum mechanics, intended to be accessible to anyone who can grok algebra and complex numbers.  Cleaning up the old confusion about QM is used to introduce basic issues in rationality (such as the technical version of Occam's Razor), epistemology, reductionism, naturalism, and philosophy of science.  Not dispensable reading, even though the exact reasons for the digression are hard to explain in advance of reading.
  The Metaethics Sequence
What words like "right" and "should" mean; how to integrate moral concepts into a naturalistic universe.
The dependencies on this sequence may not be fully organized, and the post list does not have summaries.  Yudkowsky considers this one of his less successful attempts at explanation.
The Fun Theory Sequence
A concrete theory of transhuman values.  How much fun is there in the universe; will we ever run out of fun; are we having fun yet; could we be having more fun.  Part of the complexity of value thesis.  Also forms part of the fully general answer to religious theodicy.
The Craft and the Community
The final sequence of Eliezer Yudkowsky's two-year-long string of daily posts to Less Wrong, on improving the art of rationality and building communities of rationalists.
Minor Sequences
Smaller collections of posts.  Usually parts of major sequences which depend on some-but-not-all of the points introduced.
  Map and Territory
A collection of introductory posts dealing with the fundamentals of rationality: the difference between the map and the territory, Bayes's Theorem and the nature of evidence, why anyone should care about truth, minds as reflective cognitive engines...
Seeing with Fresh Eyes
Some notes on the incredibly difficult feat of actually getting your brain to think about something (a key step in  actually changing your mind).  Whenever someone exhorts you to "think outside the box", they usually, for your convenience, point out exactly where "outside the box" is located. Isn't it funny how nonconformists all dress the same...
Subsequence of How to Actually Change Your Mind.
Politics is the Mind-Killer
Some of the various ways that politics damages our sanity - including, of course, making it harder to  change our minds on political issues.
Subsequence of How to Actually Change Your Mind.
Death Spirals and the Cult Attractor
Affective death spirals are positive feedback loops caused by the halo effect: Positive characteristics perceptually correlate, so the more nice things we say about X, the more additional nice things we're likely to believe about X.
Cultishness is an empirical attractor in human groups: roughly an affective death spiral; plus peer pressure and outcasting behavior; plus (often) defensiveness around something believed to be un-improvable.
Yet another subsequence of How to Actually Change Your Mind.
Ethical Injunctions
Ethical injunctions are rules not to do something even when you believe it's the right thing to do. This is to protect you from your own cleverness (especially taking bad black swan bets), and the Corrupted hardware you're running on.
Related to the Metaethics sequence.
Joy in the Merely Real
If dragons were common, and you could look at one in the zoo - but zebras were a rare legendary creature that had finally been decided to be mythical - then there's a certain sort of person who would ignore dragons, who would never bother to look at dragons, and chase after rumors of zebras. The grass is always greener on the other side of reality.
Which is rather setting ourselves up for eternal disappointment, eh? If we cannot take joy in the merely real, our lives shall be empty indeed.
Subsequence of  Reductionism.
  Zombies Sequence
On the putative "possibility" of beings who are just like us in every sense, but not conscious - that is, lacking inner subjective experience.
Subsequence of  Reductionism.
  The Simple Math of Evolution
Learning the  very basic math of evolutionary biology costs relatively little if you understand algebra, but gives you a surprisingly different perspective from what you'll find in strictly nonmathematical texts.
Challenging the Difficult
How to do things that are difficult or "impossible".
  Coming of Age
How Yudkowsky made epic errors of reasoning as a teenage "rationalist" and recovered from them starting at around age 23, the period that he refers to as his Bayesian Enlightenment.
Sequences by Others
Later sequences that were written people other than Eliezer Yudkowsky.
Positivism, Self Deception, and Neuroscience by Yvain
Priming and Implicit Association by Yvain
Priming may be described as the capability of any random stimulus to commandeer your thinking and judgement for the next several minutes. Scared? Don't be. There exist ways to defend yourself against these kinds of intrusions, and there are even methods to harness them into useful testing mechanisms.
  Decision Theory of Newcomblike Problems by AnnaSalamon
Decisions need to be modeled with some structure in order to be scrutinized and systematically improved; simply "intuiting" the answers to decision problems by ad-hoc methods is not conducive to thorough analysis.  For this, we formulate decision theories.  This sequence, themed with an analysis of Newcomb's problem, is a consolidated summary and context for the many decision theory discussions found on LessWrong at the time of writing.
Living Luminously by Alicorn
Luminosity, as used here, is self-awareness.  A luminous mental state is one that you have and know that you have.  It could be an emotion, a belief or alief, a disposition, a quale, a memory - anything that might happen or be stored in your brain.  What's going on in your head?
The Science of Winning at Life by lukeprog
This sequence summarizes scientifically-backed advice for "winning" at everyday life: in one's productivity, in one's relationships, in one's emotions, etc. Each post concludes with footnotes and a long list of references from the academic literature.
Rationality and Philosophy by lukeprog
This sequence explains how intuitions are used in mainstream philosophy and what the science of intuitions suggests about how intuitions should be used in philosophy.
No-Nonsense Metaethics by lukeprog
This sequence explains and defends a naturalistic approach to metaethics.
What Intelligence Tests Miss by Kaj_Sotala
A sequence summarizing the content of Keith Stanovich's book What Intelligence Tests Miss.
Why Everyone (Else) Is a Hypocrite by Kaj_Sotala
A sequence summarizing the content of Robert Kurzban's book Why Everyone (Else) is a Hypocrite: Evolution and the Modular Mind (this sequence hasn't been finished).

Q: A sequence is a series of multiple posts on Less Wrong on the same topic, to coherently and fully explore a particular thesis.
Reading the [...] is the most systematic way to approach the Less Wrong archives.
See also an alternative index of the [...], and another for people who already have a science background.
If you prefer books over blog posts, Thinking and Deciding by Jonathan Baron and Good and Real by Gary Drescher have both been mentioned as books with lots of overlap with the [...].
Contents
1 Alternative formats
2 Translations
3 Core [...]
4 Major [...]
4.1 Mysterious Answers to Mysterious Questions
4.2 A Human's Guide to Words
4.3 How To Actually Change Your Mind
4.4 Reductionism
4.5 The Quantum Physics Sequence
4.6 The Metaethics Sequence
4.7 The Fun Theory Sequence
4.8 The Craft and the Community
5 Minor [...]
5.1 Map and Territory
5.2 Seeing with Fresh Eyes
5.3 Politics is the Mind-Killer
5.4 Death Spirals and the Cult Attractor
5.5 Ethical Injunctions
5.6 Joy in the Merely Real
5.7 Zombies Sequence
5.8 The Simple Math of Evolution
5.9 Challenging the Difficult
5.10 Coming of Age
6 [...] by Others
6.1 Positivism, Self Deception, and Neuroscience by Yvain
6.2 Priming and Implicit Association by Yvain
6.3 Decision Theory of Newcomblike Problems by AnnaSalamon
6.4 Living Luminously by Alicorn
6.5 The Science of Winning at Life by lukeprog
6.6 Rationality and Philosophy by lukeprog
6.7 No-Nonsense Metaethics by lukeprog
6.8 What Intelligence Tests Miss by Kaj_Sotala
6.9 Why Everyone (Else) Is a Hypocrite by Kaj_Sotala
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Alternative formats
The [...] have been converted to eReader compatible formats by several projects.
Print ready versions by jb55 (GitHub). Has versions in Markdown, PDF, and ePub. ePubs can be converted to nearly any other format with calibre.
lw2ebook by OneWhoFrogs (GitHub). Includes all [...] in ePub and mobi formats.
[...] in EPub and MOBI formats by ciphergoth
Print ready versions by Jordan. Contains all posts of each sequence in one HTML file.
Translations
Main article: Translations into other languages
French
Italian
Spanish
Russian
Core [...]
Map and Territory contains some of the most important introductory posts and essays.
Read Mysterious Answers to Mysterious Questions and  Reductionism in order to ensure you're not making any basic rationality errors.
The most important technique that Less Wrong can offer you is How To Actually Change Your Mind.
Major [...]
Long [...] that have been completed and organized into a guide.
Mysterious Answers to Mysterious Questions
How to see through the many disguises of answers or beliefs or statements that remove curiosity without alleviating confusion.
A Human's Guide to Words
Sequence guide: 37 Ways That Words Can Be Wrong
List of posts: A Human's Guide to Words
A series on the use and abuse of words; why you can't define a word any way you like; how human brains seem to process definitions.  First introduces the Mind projection fallacy and the concept of how an algorithm feels from inside, which makes it a basic intro to key elements of the LW zeitgeist.
How To Actually Change Your Mind
A mega-sequence scattered over almost all of Less Wrong on the ultra-high-level penultimate technique of rationality: actually updating on evidence.
Organized into eight sub[...].
  Reductionism
How to take reality apart into pieces... and live in that universe, where we have always lived, without feeling disappointed about the fact that complicated things are made of simpler things.
The Quantum Physics Sequence
A non-mysterious introduction to quantum mechanics, intended to be accessible to anyone who can grok algebra and complex numbers.  Cleaning up the old confusion about QM is used to introduce basic issues in rationality (such as the technical version of Occam's Razor), epistemology, reductionism, naturalism, and philosophy of science.  Not dispensable reading, even though the exact reasons for the digression are hard to explain in advance of reading.
  The Metaethics Sequence
What words like "right" and "should" mean; how to integrate moral concepts into a naturalistic universe.
The dependencies on this sequence may not be fully organized, and the post list does not have summaries.  Yudkowsky considers this one of his less successful attempts at explanation.
The Fun Theory Sequence
A concrete theory of transhuman values.  How much fun is there in the universe; will we ever run out of fun; are we having fun yet; could we be having more fun.  Part of the complexity of value thesis.  Also forms part of the fully general answer to religious theodicy.
The Craft and the Community
The final sequence of Eliezer Yudkowsky's two-year-long string of daily posts to Less Wrong, on improving the art of rationality and building communities of rationalists.
Minor [...]
Smaller collections of posts.  Usually parts of major [...] which depend on some-but-not-all of the points introduced.
  Map and Territory
A collection of introductory posts dealing with the fundamentals of rationality: the difference between the map and the territory, Bayes's Theorem and the nature of evidence, why anyone should care about truth, minds as reflective cognitive engines...
Seeing with Fresh Eyes
Some notes on the incredibly difficult feat of actually getting your brain to think about something (a key step in  actually changing your mind).  Whenever someone exhorts you to "think outside the box", they usually, for your convenience, point out exactly where "outside the box" is located. Isn't it funny how nonconformists all dress the same...
Subsequence of How to Actually Change Your Mind.
Politics is the Mind-Killer
Some of the various ways that politics damages our sanity - including, of course, making it harder to  change our minds on political issues.
Subsequence of How to Actually Change Your Mind.
Death Spirals and the Cult Attractor
Affective death spirals are positive feedback loops caused by the halo effect: Positive characteristics perceptually correlate, so the more nice things we say about X, the more additional nice things we're likely to believe about X.
Cultishness is an empirical attractor in human groups: roughly an affective death spiral; plus peer pressure and outcasting behavior; plus (often) defensiveness around something believed to be un-improvable.
Yet another subsequence of How to Actually Change Your Mind.
Ethical Injunctions
Ethical injunctions are rules not to do something even when you believe it's the right thing to do. This is to protect you from your own cleverness (especially taking bad black swan bets), and the Corrupted hardware you're running on.
Related to the Metaethics sequence.
Joy in the Merely Real
If dragons were common, and you could look at one in the zoo - but zebras were a rare legendary creature that had finally been decided to be mythical - then there's a certain sort of person who would ignore dragons, who would never bother to look at dragons, and chase after rumors of zebras. The grass is always greener on the other side of reality.
Which is rather setting ourselves up for eternal disappointment, eh? If we cannot take joy in the merely real, our lives shall be empty indeed.
Subsequence of  Reductionism.
  Zombies Sequence
On the putative "possibility" of beings who are just like us in every sense, but not conscious - that is, lacking inner subjective experience.
Subsequence of  Reductionism.
  The Simple Math of Evolution
Learning the  very basic math of evolutionary biology costs relatively little if you understand algebra, but gives you a surprisingly different perspective from what you'll find in strictly nonmathematical texts.
Challenging the Difficult
How to do things that are difficult or "impossible".
  Coming of Age
How Yudkowsky made epic errors of reasoning as a teenage "rationalist" and recovered from them starting at around age 23, the period that he refers to as his Bayesian Enlightenment.
[...] by Others
Later [...] that were written people other than Eliezer Yudkowsky.
Positivism, Self Deception, and Neuroscience by Yvain
Priming and Implicit Association by Yvain
Priming may be described as the capability of any random stimulus to commandeer your thinking and judgement for the next several minutes. Scared? Don't be. There exist ways to defend yourself against these kinds of intrusions, and there are even methods to harness them into useful testing mechanisms.
  Decision Theory of Newcomblike Problems by AnnaSalamon
Decisions need to be modeled with some structure in order to be scrutinized and systematically improved; simply "intuiting" the answers to decision problems by ad-hoc methods is not conducive to thorough analysis.  For this, we formulate decision theories.  This sequence, themed with an analysis of Newcomb's problem, is a consolidated summary and context for the many decision theory discussions found on LessWrong at the time of writing.
Living Luminously by Alicorn
Luminosity, as used here, is self-awareness.  A luminous mental state is one that you have and know that you have.  It could be an emotion, a belief or alief, a disposition, a quale, a memory - anything that might happen or be stored in your brain.  What's going on in your head?
The Science of Winning at Life by lukeprog
This sequence summarizes scientifically-backed advice for "winning" at everyday life: in one's productivity, in one's relationships, in one's emotions, etc. Each post concludes with footnotes and a long list of references from the academic literature.
Rationality and Philosophy by lukeprog
This sequence explains how intuitions are used in mainstream philosophy and what the science of intuitions suggests about how intuitions should be used in philosophy.
No-Nonsense Metaethics by lukeprog
This sequence explains and defends a naturalistic approach to metaethics.
What Intelligence Tests Miss by Kaj_Sotala
A sequence summarizing the content of Keith Stanovich's book What Intelligence Tests Miss.
Why Everyone (Else) Is a Hypocrite by Kaj_Sotala
A sequence summarizing the content of Robert Kurzban's book Why Everyone (Else) is a Hypocrite: Evolution and the Modular Mind (this sequence hasn't been finished).
A: Sequences

Q: Shannon information
A: The Shannon entropy is a measure of the average information content one is missing when one does not know the value of the random variable.

Q: The Shannon entropy is a measure of the average information content one is missing when one does not know the value of the random variable.
A: Shannon information

Q: Shut up and multiply
A: You know what?  This isn't about your feelings.  A human life, with all its joys and all its pains, adding up over the course of decades, is worth far more than your brain's feelings of comfort or discomfort with a plan.  Does computing the expected utility feel too cold-blooded for your taste?  Well, that feeling isn't even a feather in the scales, when a life is at stake.  Just shut up and multiply.
—Circular Altruism
Due to scope neglect, framing effects, and other cognitive biases, the result of an expected utility calculation may be intuitively unappealing, perhaps even horrifying. And yet, intuition is not the most reliable guide for what policies will actually produce the best results, particularly in cases where we can actually do calculations with the relevant quantities. The ability to shut up and multiply, to trust the math even when it feels wrong is a key rationalist skill.

Q: You know what?  This isn't about your feelings.  A human life, with all its joys and all its pains, adding up over the course of decades, is worth far more than your brain's feelings of comfort or discomfort with a plan.  Does computing the expected utility feel too cold-blooded for your taste?  Well, that feeling isn't even a feather in the scales, when a life is at stake.  Just [...].
—Circular Altruism
Due to scope neglect, framing effects, and other cognitive biases, the result of an expected utility calculation may be intuitively unappealing, perhaps even horrifying. And yet, intuition is not the most reliable guide for what policies will actually produce the best results, particularly in cases where we can actually do calculations with the relevant quantities. The ability to [...], to trust the math even when it feels wrong is a key rationalist skill.
A: Shut up and multiply

Q: Signaling
A: Signaling is any activity whose purpose is to cause others to form beliefs about your abilities (whether those beliefs will be correct or not). Some signaling is performed exclusively to impress others (to improve your status), and in some cases isn't even worth that. In other cases, signaling is a side-effect of an otherwise useful activity.
For example, if doing something is easy for one type of person and hard for another type of person, you might do that thing just to get people to think you're the former type of person, even if the thing isn't in itself worth doing. This could explain many facets of human behavior, and reveal opportunities for reducing waste.
Not all signaling is about abilities. Signaling can also be about personality, current emotional state, beliefs, loyalty to a particular group, status within a group, etc.

Q: [...] is any activity whose purpose is to cause others to form beliefs about your abilities (whether those beliefs will be correct or not). Some [...] is performed exclusively to impress others (to improve your status), and in some cases isn't even worth that. In other cases, [...] is a side-effect of an otherwise useful activity.
For example, if doing something is easy for one type of person and hard for another type of person, you might do that thing just to get people to think you're the former type of person, even if the thing isn't in itself worth doing. This could explain many facets of human behavior, and reveal opportunities for reducing waste.
Not all [...] is about abilities. [...] can also be about personality, current emotional state, beliefs, loyalty to a particular group, status within a group, etc.
A: Signaling

Q: Simple math of everything
A: But for people who can read calculus, and sometimes just plain
algebra, the drop-dead basic mathematics of a field may not take that
long to learn.  And it's likely to change your outlook on life more than
the math-free popularizations or the highly technical math.
—The Simple Math of Everything
Contents
1 Computer science
1.1 Amdahl's law
1.2 Asymptotic notation
1.3 Deterministic finite state automata
1.4 The pumping lemma for regular languages
1.5 Cantor's diagonal argument
2 Links
2.1 Books
2.2 Videos
3 Blog posts
4 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Computer science
Amdahl's law
Relates the speedup of a sub-task to the resulting speedup of the whole.
Trivially true, but often needed to knock down false intuition.
on Wikipedia, long with examples
on MathWorld, short without examples
Asymptotic notation
Used to abstract away units and fixed overhead when analyzing resource
usage.
on Wikipedia, long
cheat sheet from the same article
Deterministic finite state automata
Traditional square one of theoretical computer science, with many
practical applications.
on Wikipedia, definition and example
homework with solutions (PDF)
The pumping lemma for regular languages
Illustrates many recurring themes.  Understanding the proof and usage of
the pumping lemma will help you understand and apply more famous,
advanced results (e.g. anything involving Turing Machines).
at Penn Engineering, explanation and examples
handout (PDF) with concise statement and examples
Cantor's diagonal argument
An astonishingly elegant technique for proving certain kinds of theorems. Originally introduced by the mathematician Georg Cantor to show that the set of real numbers is uncountable – that is, there is no one-to-one correspondence between real numbers and natural numbers, but was later found to generalize to several other contexts. Perhaps the most notable uses of this technique, in addition to Cantor's proof, are Alan Turing's answer to the Halting problem, and Gödel's proof of his famous first incompleteness theorem.

Q: But for people who can read calculus, and sometimes just plain
algebra, the drop-dead basic mathematics of a field may not take that
long to learn.  And it's likely to change your outlook on life more than
the math-free popularizations or the highly technical math.
—The Simple Math of Everything
Contents
1 Computer science
1.1 Amdahl's law
1.2 Asymptotic notation
1.3 Deterministic finite state automata
1.4 The pumping lemma for regular languages
1.5 Cantor's diagonal argument
2 Links
2.1 Books
2.2 Videos
3 Blog posts
4 See also
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Computer science
Amdahl's law
Relates the speedup of a sub-task to the resulting speedup of the whole.
Trivially true, but often needed to knock down false intuition.
on Wikipedia, long with examples
on MathWorld, short without examples
Asymptotic notation
Used to abstract away units and fixed overhead when analyzing resource
usage.
on Wikipedia, long
cheat sheet from the same article
Deterministic finite state automata
Traditional square one of theoretical computer science, with many
practical applications.
on Wikipedia, definition and example
homework with solutions (PDF)
The pumping lemma for regular languages
Illustrates many recurring themes.  Understanding the proof and usage of
the pumping lemma will help you understand and apply more famous,
advanced results (e.g. anything involving Turing Machines).
at Penn Engineering, explanation and examples
handout (PDF) with concise statement and examples
Cantor's diagonal argument
An astonishingly elegant technique for proving certain kinds of theorems. Originally introduced by the mathematician Georg Cantor to show that the set of real numbers is uncountable – that is, there is no one-to-one correspondence between real numbers and natural numbers, but was later found to generalize to several other contexts. Perhaps the most notable uses of this technique, in addition to Cantor's proof, are Alan Turing's answer to the Halting problem, and Gödel's proof of his famous first incompleteness theorem.
A: Simple math of everything

Q: Singularity Institute
A: The Singularity Institute is a non-profit research organization devoted to reducing existential risk from unfriendly artificial intelligence and understanding problems related to friendly artificial intelligence. Eliezer Yudkowsky was one of the early founders and continues to work there as a Research Fellow. The Singularity Institute created and currently operates Less Wrong.

Q: The [...] is a non-profit research organization devoted to reducing existential risk from unfriendly artificial intelligence and understanding problems related to friendly artificial intelligence. Eliezer Yudkowsky was one of the early founders and continues to work there as a Research Fellow. The [...] created and currently operates Less Wrong.
A: Singularity Institute

Q: Singularity paper
A: A draft page for the upcoming Less Wrong paper on the Singularity. See http://lesswrong.com/lw/1fz/a_less_wrong_singularity_article/ .
Also see: Intelligence Explosion
Contents
1 Basic structure
1.1 Abstract
1.2 The Power of Intelligence
1.3 Limitations of human intelligence, ways in which AI could be more powerful and faster
2 References
2.1 Works by Yudkowsky
2.1.1 Less Wrong posts
2.1.2 Other Yudkowsky material
2.2 Other sources and references
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
  Basic structure
Abstract
The creation of an artificial general intelligence (AGI) is plausible. Humans are the first and only example we have of a general intelligence, but we have strong empirical and theoretical reasons for believing that we're nowhere near maximally intelligent. Sufficient understanding of intelligence, or whatever similar concept can help us understand and account for human technological prowess, may allow researchers to duplicate the process in a computer. Computer-based AIs could potentially think much faster than humans, expand onto new hardware, and undergo recursive self-improvement. These possibilities suggest that such an AGI, once created, is likely to undergo an intelligence explosion, vastly increasing its capability to far beyond the human level on a very fast timescale.
The Power of Intelligence
Human civilization has had a massive impact on this planet. Humans have developed language, sophisticated tools, and advanced technology, by virtue of our general intelligence: human individuals have the ability to learn and invent new skills and teach them to other humans, as opposed to new adaptations gradually developing by natural selection. We are cultural animals, and over time, and especially since the scientific and industrial revolutions, our culture has developed more and more complex knowledge and a deeper division of labor. Once natural selection created humans with the general intelligence to sustain culture, culture took over and operates at a faster timescale than evolution.
Limitations of human intelligence, ways in which AI could be more powerful and faster
As great of an impact human intelligence has had, there are specific reasons to think it would be greatly surpassed by AI. Almost all our advances over the past thousands of years have been cultural; we haven't had time to evolve new brain architecture (cite evopsych paper? The Adapted Mind?). Evolution didn't design our brains to build civilizations; it's just sort of something that happened. We can also point to specific ways in which the human brain is lacking, where our intuitive judgement goes horribly wrong (cite heuristics and biases).
The question of FOOM
-improvement up to human-comparable programming ability
-improvement at human-comparable programming ability and beyond
-effects of hardware scale, minds that can be copied and run quickly, vs qualitative improvements
Conclusions
References
  References
List here references to work that can be used for building the argument.
Minds and Machines submission guidelines
Works by Yudkowsky
Less Wrong posts
Optimization and the Singularity
Cascades, Cycles, Insight...
...Recursion, Magic
Engelbart: Insufficiently Recursive
Total Nano Domination
Singletons Rule OK
Recursive Self-Improvement
Hard Takeoff
Permitted Possibilities, & Locality
Sustained Strong Recursion
Disjunctions, Antipredictions, Etc.
What I Think, If Not Why
Other Yudkowsky material
What Is the Singularity?
Why Work Toward the Singularity?
Three Singularity Schools
LOGI part 3, Seed AI
The Power of Intelligence
Cognitive Biases Potentially Affecting Judgement of Global Risks (PDF), in Global Catastrophic Risks
Artificial Intelligence as a Positive and Negative Factor in Global Risk (PDF), in Global Catastrophic Risks
Why We Need Friendly AI
Other sources and references
See list here  and also this

Q: A draft page for the upcoming Less Wrong paper on the Singularity. See http://lesswrong.com/lw/1fz/a_less_wrong_singularity_article/ .
Also see: Intelligence Explosion
Contents
1 Basic structure
1.1 Abstract
1.2 The Power of Intelligence
1.3 Limitations of human intelligence, ways in which AI could be more powerful and faster
2 References
2.1 Works by Yudkowsky
2.1.1 Less Wrong posts
2.1.2 Other Yudkowsky material
2.2 Other sources and references
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
  Basic structure
Abstract
The creation of an artificial general intelligence (AGI) is plausible. Humans are the first and only example we have of a general intelligence, but we have strong empirical and theoretical reasons for believing that we're nowhere near maximally intelligent. Sufficient understanding of intelligence, or whatever similar concept can help us understand and account for human technological prowess, may allow researchers to duplicate the process in a computer. Computer-based AIs could potentially think much faster than humans, expand onto new hardware, and undergo recursive self-improvement. These possibilities suggest that such an AGI, once created, is likely to undergo an intelligence explosion, vastly increasing its capability to far beyond the human level on a very fast timescale.
The Power of Intelligence
Human civilization has had a massive impact on this planet. Humans have developed language, sophisticated tools, and advanced technology, by virtue of our general intelligence: human individuals have the ability to learn and invent new skills and teach them to other humans, as opposed to new adaptations gradually developing by natural selection. We are cultural animals, and over time, and especially since the scientific and industrial revolutions, our culture has developed more and more complex knowledge and a deeper division of labor. Once natural selection created humans with the general intelligence to sustain culture, culture took over and operates at a faster timescale than evolution.
Limitations of human intelligence, ways in which AI could be more powerful and faster
As great of an impact human intelligence has had, there are specific reasons to think it would be greatly surpassed by AI. Almost all our advances over the past thousands of years have been cultural; we haven't had time to evolve new brain architecture (cite evopsych paper? The Adapted Mind?). Evolution didn't design our brains to build civilizations; it's just sort of something that happened. We can also point to specific ways in which the human brain is lacking, where our intuitive judgement goes horribly wrong (cite heuristics and biases).
The question of FOOM
-improvement up to human-comparable programming ability
-improvement at human-comparable programming ability and beyond
-effects of hardware scale, minds that can be copied and run quickly, vs qualitative improvements
Conclusions
References
  References
List here references to work that can be used for building the argument.
Minds and Machines submission guidelines
Works by Yudkowsky
Less Wrong posts
Optimization and the Singularity
Cascades, Cycles, Insight...
...Recursion, Magic
Engelbart: Insufficiently Recursive
Total Nano Domination
Singletons Rule OK
Recursive Self-Improvement
Hard Takeoff
Permitted Possibilities, & Locality
Sustained Strong Recursion
Disjunctions, Antipredictions, Etc.
What I Think, If Not Why
Other Yudkowsky material
What Is the Singularity?
Why Work Toward the Singularity?
Three Singularity Schools
LOGI part 3, Seed AI
The Power of Intelligence
Cognitive Biases Potentially Affecting Judgement of Global Risks (PDF), in Global Catastrophic Risks
Artificial Intelligence as a Positive and Negative Factor in Global Risk (PDF), in Global Catastrophic Risks
Why We Need Friendly AI
Other sources and references
See list here  and also this
A: Singularity paper

Q: Slowness of evolution
A: Suppose a beneficial mutation which conveys a fitness advantage of 3%: on average, bearers of this gene have 1.03 times as many children as non-bearers.  Spreading through a population of 100,000, such a gene would require an average of 768 generations to reach universality in the gene pool.  A population of 500,000 would require 875 generations.  The general formula is
   * Generations to fixation = 2 ln(N) / s
where N is the population size, and (1 + s) is the fitness.  (If each bearer of the gene has 1.03 times as many children as a non-bearer, s = 0.03.)
Thus, if N were 1,000,000 - the estimated global population in hunter-gatherer times - it would require 2763 generations for a gene conveying a 1% advantage to spread through the gene pool.
All this assumes that the gene spreads in the first place.  Here the equation is simpler and ends up not depending at all on population size:
   * Probability of fixation = 2s
A mutation conveying a 3% advantage (which is pretty darned large, as mutations go) has a 6% chance of spreading, at least on that occasion.  Mutations can happen more than once, but in a population of a million with a copying fidelity of 10^-8 errors per base per generation, you may have to wait a hundred generations for another (6%) chance.
Complex, interdependent adaptations take a very long time to evolve.  First comes allele A, which is advantageous of itself, and requires a thousand generations to fixate in the gene pool.  Only then can another allele B, which depends on A, begin rising to fixation.  Suppose B confers a 5% advantage in the presence of A, no advantage otherwise.  Then while A is still at 1% frequency in the population, B only confers its advantage 1 out of 100 times, so the average fitness advantage of B is 0.05%, and B's probability of fixation is 0.1%.
Contrast to a human programmer, who can design a new complex mechanism with a hundred interdependent parts over the course of a single afternoon.
The tremendously slow timescale of evolution, especially for creating new complex machinery (as opposed to selecting on existing variance), is why the behavior of evolved organisms is often better interpreted in terms of what did in fact work yesterday, rather than what will work in the future.

Q: Suppose a beneficial mutation which conveys a fitness advantage of 3%: on average, bearers of this gene have 1.03 times as many children as non-bearers.  Spreading through a population of 100,000, such a gene would require an average of 768 generations to reach universality in the gene pool.  A population of 500,000 would require 875 generations.  The general formula is
   * Generations to fixation = 2 ln(N) / s
where N is the population size, and (1 + s) is the fitness.  (If each bearer of the gene has 1.03 times as many children as a non-bearer, s = 0.03.)
Thus, if N were 1,000,000 - the estimated global population in hunter-gatherer times - it would require 2763 generations for a gene conveying a 1% advantage to spread through the gene pool.
All this assumes that the gene spreads in the first place.  Here the equation is simpler and ends up not depending at all on population size:
   * Probability of fixation = 2s
A mutation conveying a 3% advantage (which is pretty darned large, as mutations go) has a 6% chance of spreading, at least on that occasion.  Mutations can happen more than once, but in a population of a million with a copying fidelity of 10^-8 errors per base per generation, you may have to wait a hundred generations for another (6%) chance.
Complex, interdependent adaptations take a very long time to evolve.  First comes allele A, which is advantageous of itself, and requires a thousand generations to fixate in the gene pool.  Only then can another allele B, which depends on A, begin rising to fixation.  Suppose B confers a 5% advantage in the presence of A, no advantage otherwise.  Then while A is still at 1% frequency in the population, B only confers its advantage 1 out of 100 times, so the average fitness advantage of B is 0.05%, and B's probability of fixation is 0.1%.
Contrast to a human programmer, who can design a new complex mechanism with a hundred interdependent parts over the course of a single afternoon.
The tremendously slow timescale of evolution, especially for creating new complex machinery (as opposed to selecting on existing variance), is why the behavior of evolved organisms is often better interpreted in terms of what did in fact work yesterday, rather than what will work in the future.
A: Slowness of evolution

Q: Solomonoff induction
A: Ray Solomonoff defined an inference
system that will learn to correctly predict any computable
sequence with only the absolute minimum amount of data.  This
system, in a certain sense, is the perfect universal prediction
algorithm.  To summarize it very informally, Solomonoff induction
works by:
Starting with all possible hypotheses (sequences) as represented by computer programs (that generate those sequences), weighted by their simplicity (2-n, where n is the program length);
Discarding those hypotheses that are inconsistent with the data.
Weighting hypotheses by simplicity, the system automatically
incorporates a form of Occam's razor, which is why it has been
playfully referred to as Solomonoff's lightsaber.
Solomonoff induction gets off the ground with a solution to the
"problem of the priors".  Suppose that you stand before a
universal prefix Turing machine
U.  You are interested in a certain finite output
string y0.  In particular, you want to know the
probability that U will produce the output
y0 given a random input tape.  This probability is
the Solomonoff a priori probability of
y0.
More precisely, suppose that a particular infinite input string
x0 is about to be fed into
U.  However, you know nothing about
x0 other than that each term of the string is either
0 or 1.  As far as your
state of knowledge is concerned, the ith digit of
x0 is as likely to be 0 as it is to
be 1, for all .  You
want to find the a priori probability m(y0) of
the following proposition:
(*) If U takes in x0
as input, then U will produce output y0
and then halt.
Unfortunately, computing the exact value of m(y0) would require solving the halting problem, which is undecidable.  Nonetheless, it is easy to derive an expression for m(y0).  If U halts on an infinite input string x, then U must read only a finite initial segment of x, after which U immediately halts.  We call a finite string p a self-delimiting program if and only if there exists an infinite input string x beginning with p such that U halts on x immediately after reading to the end of p.  The set  of self-delimiting programs is the prefix code for U.  It is the determination of the elements of  that requires a solution to the halting problem.
Given , we write "" to express the proposition that x0 begins with p, and we write "U(p) = y0" to express the proposition that U produces output y0, and then halts, when fed any input beginning with p.  Proposition (*) is then equivalent to the exclusive disjunction
Since x0 was chosen at random from {0,1}ω, we take the probability of  to be , where  is the length of p as a bit string.  Hence, the probability of (*) is

Q: Ray Solomonoff defined an inference
system that will learn to correctly predict any computable
sequence with only the absolute minimum amount of data.  This
system, in a certain sense, is the perfect universal prediction
algorithm.  To summarize it very informally, [...]
works by:
Starting with all possible hypotheses (sequences) as represented by computer programs (that generate those sequences), weighted by their simplicity (2-n, where n is the program length);
Discarding those hypotheses that are inconsistent with the data.
Weighting hypotheses by simplicity, the system automatically
incorporates a form of Occam's razor, which is why it has been
playfully referred to as Solomonoff's lightsaber.
[...] gets off the ground with a solution to the
"problem of the priors".  Suppose that you stand before a
universal prefix Turing machine
U.  You are interested in a certain finite output
string y0.  In particular, you want to know the
probability that U will produce the output
y0 given a random input tape.  This probability is
the Solomonoff a priori probability of
y0.
More precisely, suppose that a particular infinite input string
x0 is about to be fed into
U.  However, you know nothing about
x0 other than that each term of the string is either
0 or 1.  As far as your
state of knowledge is concerned, the ith digit of
x0 is as likely to be 0 as it is to
be 1, for all .  You
want to find the a priori probability m(y0) of
the following proposition:
(*) If U takes in x0
as input, then U will produce output y0
and then halt.
Unfortunately, computing the exact value of m(y0) would require solving the halting problem, which is undecidable.  Nonetheless, it is easy to derive an expression for m(y0).  If U halts on an infinite input string x, then U must read only a finite initial segment of x, after which U immediately halts.  We call a finite string p a self-delimiting program if and only if there exists an infinite input string x beginning with p such that U halts on x immediately after reading to the end of p.  The set  of self-delimiting programs is the prefix code for U.  It is the determination of the elements of  that requires a solution to the halting problem.
Given , we write "" to express the proposition that x0 begins with p, and we write "U(p) = y0" to express the proposition that U produces output y0, and then halts, when fed any input beginning with p.  Proposition (*) is then equivalent to the exclusive disjunction
Since x0 was chosen at random from {0,1}ω, we take the probability of  to be , where  is the length of p as a bit string.  Hence, the probability of (*) is
A: Solomonoff induction

Q: Something to protect
A: The Art must have a purpose other than itself, or it collapses into infinite recursion.

Q: The Art must have a purpose other than itself, or it collapses into infinite recursion.
A: Something to protect

Q: Sound argument
A: Sound argument is argument that is valid and whose premises are all true. In other words, the premises are true and the conclusion necessarily follows from them, making the conclusion true as well.
For example, consider the following syllogism:

Q: [...] is argument that is valid and whose premises are all true. In other words, the premises are true and the conclusion necessarily follows from them, making the conclusion true as well.
For example, consider the following syllogism:
A: Sound argument

Q: Spaced repetition
A: Spaced repetition (often abbreviated SR) is a technique for building long-term knowledge efficiently. The technique works by showing you a flash card just before a computer model predicts you will have forgotten it.
Anki is Less Wrong's spaced repetition software of choice.
Contents
1 Literature
2 Blog posts
3 Supermemo material
4 SR decks
4.1 SR cards for Sequences
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Literature
gwern's spaced literature repetition review won a contest for the best literature review of spaced repetition (includes anki cards for gwern's post). Check out the summary for some quick tips.
Blog posts
Self-programming through spaced repetition by Luke_Grecki
Cheat codes by sketerpot
I want a better memory by alexflint
SRS advice by Dr_Manhattan
Anki on Android in 60 seconds, Anki on Mac in 60 seconds by lukeprog
All Less Wrong posts tagged "spaced_repetition"
Supermemo material
Many articles on assorted related topics
Research on memory and learning
Frequently asked questions about various aspects of spaced repetition
SR decks
Decks (links, or for Anki, the names of a deck in the Anki collection) relevant to LW.

Q: [...] (often abbreviated SR) is a technique for building long-term knowledge efficiently. The technique works by showing you a flash card just before a computer model predicts you will have forgotten it.
Anki is Less Wrong's [...] software of choice.
Contents
1 Literature
2 Blog posts
3 Supermemo material
4 SR decks
4.1 SR cards for Sequences
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Literature
gwern's spaced literature repetition review won a contest for the best literature review of [...] (includes anki cards for gwern's post). Check out the summary for some quick tips.
Blog posts
Self-programming through [...] by Luke_Grecki
Cheat codes by sketerpot
I want a better memory by alexflint
SRS advice by Dr_Manhattan
Anki on Android in 60 seconds, Anki on Mac in 60 seconds by lukeprog
All Less Wrong posts tagged "spaced_repetition"
Supermemo material
Many articles on assorted related topics
Research on memory and learning
Frequently asked questions about various aspects of [...]
SR decks
Decks (links, or for Anki, the names of a deck in the Anki collection) relevant to LW.
A: Spaced repetition

Q: Speed limit and complexity bound for evolution
A: (This page summarizes a discussion on the blog.)
A Yudkowsky post of November 4th, 2007, Natural selection's speed limit and complexity bound, tried to argue mathematically that there could be at most 25MB of meaningful information (or thereabouts) in the human genome, but computer simulations failed to bear out the mathematical argument.  It does seem probable that evolution has some kind of speed limit and complexity bound - eminent evolutionary biologists seem to believe it, and in fact the Genome Project discovered only 25,000 genes in the human genome - but this particular math may not be the correct argument.
The first lemma of Yudkowsky's argument was the idea that if, say, 2 parents have 16 children, and on average only 2 of those children survive, then 1 out of 8 children survive, which corresponds to 3 bits of information-theoretical information.  This part of the argument is taken from R. P. Worden's paper A Speed Limit For Evolution[1] but would not seem to agree with the computer simulation in question, so it is possible that Worden imposed additional conditions (or perhaps the paper itself is wrong).  According to Worden's paper, this is an evolutionary limit on the whole species - if the average surviving child is part of a litter of 16, then the 3-bit bound on information accumulated is not per couple but for the species as a whole.  In general Worden speaks of a species accumulating at most O(1) bits of information per generation.
The second lemma of Yudkowsky's argument is a well-known principle known as "one mutation, one death" which states that deleterious mutations (and the vast majority of mutations are deleterious) cause an equal number of deaths in the gene pool, whether the mutation is very harmful to an individual or only slightly harmful.  At equilibrium, deleterious mutations must be eliminated from the gene pool at the same rate they are introduced: each event in which a copying error creates a mutation, must be eliminated by an extra death of an individual bearing that mutation.  If a mutation is only very slightly deleterious - if it only kills one out of ten thousand bearers, say (or prevents one out of ten thousand children from being born) - then the mutation will spread farther before causing the deaths that prevent the mutation from spreading further.  (This is a very disheartening Malthusian principle in general; if you invent glasses to make nearsightedness less dangerous, then more people will become nearsighted and the total danger will go back up.)
From the "one mutation, one death" lemma, Yudkowsky argued that each meaningful DNA base would require around the same amount of selection pressure to support its continued existence.  Worden calculates a selection pressure of O(1) bits per generation in mammals, and the mutation rate in mammals is 10^-8 errors per base per generation.  From this, Yudkowsky argued that at most 10^8 DNA bases = 25 megabytes of meaningful information, could be sustained by mammalian evolution against the degenerative pressure of mutation.
The idea of an upper bound on the sustainable information in a genome, and that mammals are already at this upper bound and have probably been there for tens of millions of years if not longer, is not original to Yudkowsky; it is found for example in George Williams's Adaptation and Natural Selection[2] (and duly credited).  Indeed the essential idea goes back to Kimura, or even Fisher.  Yudkowsky's novelty was to attempt to calculate the bound.
Although the actual Genome Project's finding of 25,000 genes fits well under Yudkowsky's attempted bound, the mathematical argument failed.  A computer simulation failed to bear out the bound, and the flaw appears to have been as follows:  Even if one mutation creates one death, this does not mean that one death eliminates only a single mutation.  Organisms bearing more deleterious mutations are more likely to lose the evolutionary competition, and so each death can eliminate more mutations than average.  If mating is random and the least fit organisms are perfectly eliminated in every generation, the information supportable in the genome goes as the inverse square of the mutation rate.
(Why?  This may be a bit difficult to visualize.  Roughly, if the average number of deleterious mutations in the gene pool is N, and mating is random, then the difference between the average number of mutations in the 'upper half' of the population, and the number of mutations in the 'lower half' of the population, is around sqrt(N).  So eliminating the lower half of the population decreases the average number of mutations by sqrt(N).  So at equilibrium, the mutation rate can introduce sqrt(N) mutations and be eliminated from the gene pool at the same rate, with N mutations total.  Thus the supportable information goes as the inverse square of the mutation rate, under these assumptions and as borne out by computer simulation.  A similar argument shows that with many beneficial mutations being introduced, and with random mating and perfect selection, information can be gained in the genome at a rate which goes as the square root of the number of incoming beneficial mutations (which would go as the square root of the genome size).  It is possible that Worden made other assumptions which rule out this scenario.)
On the whole, the state of the argument can be described as follows:  Fisher proposed that there would be a limit to how much selection a population could sustain without going extinct.  Kimura suggested that mutation rates would determine the sustainable genetic information.  Williams argued that most organisms would long since have hit this upper bound for their species, and that there was no reason to believe that mammals were more complex than e.g. dinosaurs.  The actual Human Genome Project found only 25,000 protein-coding genes in the human genome, and humans are around 95% genetically similar to chimpanzees after five million years of divergence.  Worden calculated O(1) bits of information absorbable per generation (which must either have been wrong, or required some additional assumptions).  Yudkowsky's attempt to calculate an upper bound of 25MB of information in the human genome was contradicted by computer simulation, but this does not mean the actual information is greater than this.  If we assume that only Yudkowsky's attempt at a novel contribution actually failed and that we should go on trusting the other authors, then on the whole it is still safe to say that evolutionary biologists think there are speed limits and complexity bounds - just not that the complexity bound is 25MB calculated the way Yudkowsky tried to calculate it.

Q: (This page summarizes a discussion on the blog.)
A Yudkowsky post of November 4th, 2007, Natural selection's speed limit and complexity bound, tried to argue mathematically that there could be at most 25MB of meaningful information (or thereabouts) in the human genome, but computer simulations failed to bear out the mathematical argument.  It does seem probable that evolution has some kind of speed limit and complexity bound - eminent evolutionary biologists seem to believe it, and in fact the Genome Project discovered only 25,000 genes in the human genome - but this particular math may not be the correct argument.
The first lemma of Yudkowsky's argument was the idea that if, say, 2 parents have 16 children, and on average only 2 of those children survive, then 1 out of 8 children survive, which corresponds to 3 bits of information-theoretical information.  This part of the argument is taken from R. P. Worden's paper A Speed Limit For Evolution[1] but would not seem to agree with the computer simulation in question, so it is possible that Worden imposed additional conditions (or perhaps the paper itself is wrong).  According to Worden's paper, this is an evolutionary limit on the whole species - if the average surviving child is part of a litter of 16, then the 3-bit bound on information accumulated is not per couple but for the species as a whole.  In general Worden speaks of a species accumulating at most O(1) bits of information per generation.
The second lemma of Yudkowsky's argument is a well-known principle known as "one mutation, one death" which states that deleterious mutations (and the vast majority of mutations are deleterious) cause an equal number of deaths in the gene pool, whether the mutation is very harmful to an individual or only slightly harmful.  At equilibrium, deleterious mutations must be eliminated from the gene pool at the same rate they are introduced: each event in which a copying error creates a mutation, must be eliminated by an extra death of an individual bearing that mutation.  If a mutation is only very slightly deleterious - if it only kills one out of ten thousand bearers, say (or prevents one out of ten thousand children from being born) - then the mutation will spread farther before causing the deaths that prevent the mutation from spreading further.  (This is a very disheartening Malthusian principle in general; if you invent glasses to make nearsightedness less dangerous, then more people will become nearsighted and the total danger will go back up.)
From the "one mutation, one death" lemma, Yudkowsky argued that each meaningful DNA base would require around the same amount of selection pressure to support its continued existence.  Worden calculates a selection pressure of O(1) bits per generation in mammals, and the mutation rate in mammals is 10^-8 errors per base per generation.  From this, Yudkowsky argued that at most 10^8 DNA bases = 25 megabytes of meaningful information, could be sustained by mammalian evolution against the degenerative pressure of mutation.
The idea of an upper bound on the sustainable information in a genome, and that mammals are already at this upper bound and have probably been there for tens of millions of years if not longer, is not original to Yudkowsky; it is found for example in George Williams's Adaptation and Natural Selection[2] (and duly credited).  Indeed the essential idea goes back to Kimura, or even Fisher.  Yudkowsky's novelty was to attempt to calculate the bound.
Although the actual Genome Project's finding of 25,000 genes fits well under Yudkowsky's attempted bound, the mathematical argument failed.  A computer simulation failed to bear out the bound, and the flaw appears to have been as follows:  Even if one mutation creates one death, this does not mean that one death eliminates only a single mutation.  Organisms bearing more deleterious mutations are more likely to lose the evolutionary competition, and so each death can eliminate more mutations than average.  If mating is random and the least fit organisms are perfectly eliminated in every generation, the information supportable in the genome goes as the inverse square of the mutation rate.
(Why?  This may be a bit difficult to visualize.  Roughly, if the average number of deleterious mutations in the gene pool is N, and mating is random, then the difference between the average number of mutations in the 'upper half' of the population, and the number of mutations in the 'lower half' of the population, is around sqrt(N).  So eliminating the lower half of the population decreases the average number of mutations by sqrt(N).  So at equilibrium, the mutation rate can introduce sqrt(N) mutations and be eliminated from the gene pool at the same rate, with N mutations total.  Thus the supportable information goes as the inverse square of the mutation rate, under these assumptions and as borne out by computer simulation.  A similar argument shows that with many beneficial mutations being introduced, and with random mating and perfect selection, information can be gained in the genome at a rate which goes as the square root of the number of incoming beneficial mutations (which would go as the square root of the genome size).  It is possible that Worden made other assumptions which rule out this scenario.)
On the whole, the state of the argument can be described as follows:  Fisher proposed that there would be a limit to how much selection a population could sustain without going extinct.  Kimura suggested that mutation rates would determine the sustainable genetic information.  Williams argued that most organisms would long since have hit this upper bound for their species, and that there was no reason to believe that mammals were more complex than e.g. dinosaurs.  The actual Human Genome Project found only 25,000 protein-coding genes in the human genome, and humans are around 95% genetically similar to chimpanzees after five million years of divergence.  Worden calculated O(1) bits of information absorbable per generation (which must either have been wrong, or required some additional assumptions).  Yudkowsky's attempt to calculate an upper bound of 25MB of information in the human genome was contradicted by computer simulation, but this does not mean the actual information is greater than this.  If we assume that only Yudkowsky's attempt at a novel contribution actually failed and that we should go on trusting the other authors, then on the whole it is still safe to say that evolutionary biologists think there are speed limits and complexity bounds - just not that the complexity bound is 25MB calculated the way Yudkowsky tried to calculate it.
A: Speed limit and complexity bound for evolution

Q: Standard of evidence
A: A standard of evidence or standard of admissible evidence seeks to narrow down the type of rational evidence which may be brought to bear on any kind of (partially or wholly) adversarial process, such as scientific research, law or politics.  A common standard of evidence is thought to be needed in any such process, so as to avoid a breakdown of useful deliberation.   This is the basic contribution of the scientific method and of legal evidence standards.

Q: A [...] or standard of admissible evidence seeks to narrow down the type of rational evidence which may be brought to bear on any kind of (partially or wholly) adversarial process, such as scientific research, law or politics.  A common [...] is thought to be needed in any such process, so as to avoid a breakdown of useful deliberation.   This is the basic contribution of the scientific method and of legal evidence standards.
A: Standard of evidence

Q: Statistical bias
A: Bias as used in the field of statistics refers to directional error in an estimator.  Statistical bias is error you cannot correct by repeating the experiment many times and averaging together the results.

Q: Bias as used in the field of statistics refers to directional error in an estimator.  [...] is error you cannot correct by repeating the experiment many times and averaging together the results.
A: Statistical bias

Q: Status
A: Real or perceived relative measure of social standing, which is a function of both resource control and how one is viewed by others.

Q: Real or perceived relative measure of social standing, which is a function of both resource control and how one is viewed by others.
A: Status

Q: Status quo bias
A: The status quo bias is a cognitive bias for the status quo; in other words, people tend to avoid changing the established behavior or beliefs unless the pressure to change is sufficiently strong.
The reversal test is a technique for recognizing fallacious counterarguments against change. If the counterargument states that the change of some parameter in one direction is undesirable, the reversal test is to check whether either the change of that parameter in the opposite direction (away from status quo) is desirable, or that there are strong reasons to expect that the current value of the parameter is (at least locally) the optimal one.

Q: The [...] is a cognitive bias for the status quo; in other words, people tend to avoid changing the established behavior or beliefs unless the pressure to change is sufficiently strong.
The reversal test is a technique for recognizing fallacious counterarguments against change. If the counterargument states that the change of some parameter in one direction is undesirable, the reversal test is to check whether either the change of that parameter in the opposite direction (away from status quo) is desirable, or that there are strong reasons to expect that the current value of the parameter is (at least locally) the optimal one.
A: Status quo bias

Q: Stupidity
A: Stupidity is a word which is generally used to mean low intelligence.  However, it has other, more specialized definitions which may be more appropriate to Less Wrong's goals:
Carlo Maria Cipolla defines stupidity as the tendency to commit acts that damage others while not benefiting, or even harming, oneself.
A common definition of stupidity is "doing the same thing over and over again, while expecting different results".
According to Robert J. Sternberg, stupidity is widespread in politics, despite the fact that politicians generally have quite high intelligence.
Contrast: winning

Q: [...] is a word which is generally used to mean low intelligence.  However, it has other, more specialized definitions which may be more appropriate to Less Wrong's goals:
Carlo Maria Cipolla defines [...] as the tendency to commit acts that damage others while not benefiting, or even harming, oneself.
A common definition of [...] is "doing the same thing over and over again, while expecting different results".
According to Robert J. Sternberg, [...] is widespread in politics, despite the fact that politicians generally have quite high intelligence.
Contrast: winning
A: Stupidity

Q: Stupidity of evolution
A: Evolution can only access a very limited area in the design space, and can only search for the new designs very slowly, for a variety of reasons. The wonder of evolution is not how intelligently it works, but that an accidentally occurring optimizer without a brain works at all.

Q: Evolution can only access a very limited area in the design space, and can only search for the new designs very slowly, for a variety of reasons. The wonder of evolution is not how intelligently it works, but that an accidentally occurring optimizer without a brain works at all.
A: Stupidity of evolution

Q: Superexponential conceptspace
A: In order to do inference, we constantly need to make use of categories and concepts: it is neither possible nor desirable to deal with every unique arrangement of quarks and leptons on an individual basis. Fortunately, we can talk about repeatable higher-level regularities in the world instead: we can distinguish particular configurations of matter as instantiations of object concepts like chair or human, and say that these objects have particular properties, like red or alive.
The sheer number of distinct configurations in which matter could be arranged is unimaginably vast, but the superexponential conceptspace of the number of different ways to categorize these possible objects is even vaster.
For example, given an object that can either have or not have each of n properties, there are 2^n
different descriptions corresponding to the possible objects of that kind (a number exponential in n). The number of possible concepts, each of which either includes a given description or doesn't, is one exponential higher:
2^(2^n)
Without an inductive bias, restricting attention to only a small portion of possible concepts, it's not possible to navigate the conceptspace: to learn a concept, a "fully general" learner would need to see all the individual examples that define it. Using probability to mark the extent to which each possibility belongs to a concept is another approach to express prior information and its control over the process of learning.

Q: In order to do inference, we constantly need to make use of categories and concepts: it is neither possible nor desirable to deal with every unique arrangement of quarks and leptons on an individual basis. Fortunately, we can talk about repeatable higher-level regularities in the world instead: we can distinguish particular configurations of matter as instantiations of object concepts like chair or human, and say that these objects have particular properties, like red or alive.
The sheer number of distinct configurations in which matter could be arranged is unimaginably vast, but the [...] of the number of different ways to categorize these possible objects is even vaster.
For example, given an object that can either have or not have each of n properties, there are 2^n
different descriptions corresponding to the possible objects of that kind (a number exponential in n). The number of possible concepts, each of which either includes a given description or doesn't, is one exponential higher:
2^(2^n)
Without an inductive bias, restricting attention to only a small portion of possible concepts, it's not possible to navigate the conceptspace: to learn a concept, a "fully general" learner would need to see all the individual examples that define it. Using probability to mark the extent to which each possibility belongs to a concept is another approach to express prior information and its control over the process of learning.
A: Superexponential conceptspace

Q: Superstimulus
A: A candy bar is a superstimulus: it contains more concentrated sugar, salt, and fat than anything that exists in the ancestral environment. A candy bar matches taste buds that evolved in a hunter-gatherer environment, but it matches those taste buds much more strongly than anything that actually existed in the hunter-gatherer environment. The signal that once reliably correlated to healthy food has been hijacked, blotted out with a point in tastespace that wasn't in the training dataset - an impossibly distant outlier on the old ancestral graphs. Tastiness, formerly representing the evolutionarily identified correlates of healthiness, has been reverse-engineered and perfectly matched with an artificial substance. Unfortunately there's no equally powerful market incentive to make the resulting food item as healthy as it is tasty. We can't taste healthfulness, after all.

Q: A candy bar is a [...]: it contains more concentrated sugar, salt, and fat than anything that exists in the ancestral environment. A candy bar matches taste buds that evolved in a hunter-gatherer environment, but it matches those taste buds much more strongly than anything that actually existed in the hunter-gatherer environment. The signal that once reliably correlated to healthy food has been hijacked, blotted out with a point in tastespace that wasn't in the training dataset - an impossibly distant outlier on the old ancestral graphs. Tastiness, formerly representing the evolutionarily identified correlates of healthiness, has been reverse-engineered and perfectly matched with an artificial substance. Unfortunately there's no equally powerful market incentive to make the resulting food item as healthy as it is tasty. We can't taste healthfulness, after all.
A: Superstimulus

Q: Surprise
A: Recognizing a fact that disagrees with your intuition as surprising is an important step in updating your worldview.

Q: Recognizing a fact that disagrees with your intuition as surprising is an important step in updating your worldview.
A: Surprise

Q: Sympathetic magic
A: Humans seem to naturally generate a series of concepts known as sympathetic magic, a host of theories and practices which have certain principles in common, two of which are of overriding importance: the Law of Contagion holds that two things which have interacted, or were once part of a single entity, retain their connection and can exert influence over each other; the Law of Similarity holds that things which are similar or treated the same establish a connection and can affect each other.

Q: Humans seem to naturally generate a series of concepts known as [...], a host of theories and practices which have certain principles in common, two of which are of overriding importance: the Law of Contagion holds that two things which have interacted, or were once part of a single entity, retain their connection and can exert influence over each other; the Law of Similarity holds that things which are similar or treated the same establish a connection and can affect each other.
A: Sympathetic magic

Q: Tapping Out
A: The appropriate way to signal that you've said all you wanted to say on a particular topic, and that you're ending your participation in a conversation lest you start saying things that are less worthwhile. The Less Wrong tradition was started by Rain in this comment.
Tapping out doesn't mean accepting defeat or claiming victory, and it doesn't mean you get the last word. It just means that you don't expect your further comments in a thread to be worthwhile, because you've already made all the points you wanted to, or because you find yourself getting too emotionally invested, or for any other reason you find suitable.
The analogy comes from the practice of "tapping out" from a sparring match when one is tired, or at risk of injury, or has simply had one's fill.

Q: The appropriate way to signal that you've said all you wanted to say on a particular topic, and that you're ending your participation in a conversation lest you start saying things that are less worthwhile. The Less Wrong tradition was started by Rain in this comment.
Tapping out doesn't mean accepting defeat or claiming victory, and it doesn't mean you get the last word. It just means that you don't expect your further comments in a thread to be worthwhile, because you've already made all the points you wanted to, or because you find yourself getting too emotionally invested, or for any other reason you find suitable.
The analogy comes from the practice of "[...]" from a sparring match when one is tired, or at risk of injury, or has simply had one's fill.
A: Tapping Out

Q: Technical explanation
A: A technical explanation is an explanation of a phenomenon that makes you anticipate certain experiences. A proper technical explanation controls anticipation strictly, weighting your priors and evidence precisely to create the justified amount of uncertainty.
Technical explanations are contrasted with verbal explanations, which give the impression of understanding without actually producing the proper expectation.

Q: A [...] is an explanation of a phenomenon that makes you anticipate certain experiences. A proper [...] controls anticipation strictly, weighting your priors and evidence precisely to create the justified amount of uncertainty.
[...]s are contrasted with verbal explanations, which give the impression of understanding without actually producing the proper expectation.
A: Technical explanation

Q: Teleology
A: The study of things that happen for the sake of their future consequences.
Since nothing is ever actually the causal result of a future event, 'teleology' names a fallacy, that is, the fallacy of reasoning as if an event could be the result of a future event, its consequence.
A non-fallacious interpretation of teleology is as the study of things that happen because of their intended results, where the intention existed in an actual mind in the prior past, and so was causally able to bring about the event by planning and acting.

Q: The study of things that happen for the sake of their future consequences.
Since nothing is ever actually the causal result of a future event, '[...]' names a fallacy, that is, the fallacy of reasoning as if an event could be the result of a future event, its consequence.
A non-fallacious interpretation of [...] is as the study of things that happen because of their intended results, where the intention existed in an actual mind in the prior past, and so was causally able to bring about the event by planning and acting.
A: Teleology

Q: Terminal value
A: A terminal value (also known as an intrinsic value) is an ultimate goal, an end-in-itself.
In an artificial general intelligence with a utility or reward function, the terminal value is the maximization of that function. The non-standard term  "supergoal" is used for this concept in Eliezer Yudkowsky's earlier writings.
Contents
1 Terminal vs. instrumental vales
2 Human terminal values
3 Non-human terminal values
4 In a Friendly AI
5 Links
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Terminal vs. instrumental vales
Terminal values stand in contrast to instrumental values (also known as extrinsic values), which are means-to-an-end, mere tools in achieving terminal values. For example, if a given university student studies merely as a professional qualification, his terminal value is getting a job, while getting good grades is an instrument to that end. If a (simple) chess program tries to maximize piece value three turns into the future, that is an instrumental value to its terminal value of winning the game.
Some values may be called "terminal" merely in relation to an instrumental goal, yet themselves serve instrumentally towards a higher goal. However, in considering future artificial general intelligence, the phrase "terminal value" is  generally used only for the top level of the goal hierarchy: the true ultimate goals of a system, those which do not serve any higher value.
Human terminal values
Humans' system of terminal values is quite complex. The values were forged by evolution in the ancestral environment to maximize inclusive genetic fitness. These values include survival, health, friendship, social status, love, joy, aesthetic pleasure, curiosity, and much more. Evolution's implicit goal is inclusive genetic fitness, but humans do not have inclusive genetic fitness as a goal. Rather, these  values, which were instrumental to inclusive genetic fitness, have become humans' terminal values (an example of subgoal stomp).
Humans cannot fully introspect their terminal values. Humans' terminal values are often mutually contradictory, inconsistent, and change over time.
Non-human terminal values
Future artificial general intelligences may have the maximization of a utility function or of a reward function (reinforcement learning) as their terminal value.  The function will likely be set by the AGI's designers.
Since people make tools instrumentally, to serve specific human values, the assigned value system of the artificial general intelligence may be much simpler than humans'. This will pose a danger, as an AI must seek to protect all human values if a positive human future is to be achieved. The paperclip maximizer is a thought experiment about an artificial general intelligence with consequences disastrous to humanity, with the the apparently innocuous terminal value of maximizing the number of paperclips in its collection,
An intelligence can work towards any terminal value, not just human-like ones. AIXI is a mathematical formalism for modeling intelligence. It illustrates that the arbitrariness of terminal values may be optimized by an intelligence: AIXI  is provably more intelligent than any other agent for any computable reward function.
In a Friendly AI
For an artificial general intelligence to have a positive and not a negative effect on humanity, its terminal value must be benevolent to humans. It must seek the maximization of the full set of human values (for the humans' benefit, not for itself).
Links
Eliezer Yudkowsky, Terminal Values and Instrumental Values

Q: A [...] (also known as an intrinsic value) is an ultimate goal, an end-in-itself.
In an artificial general intelligence with a utility or reward function, the [...] is the maximization of that function. The non-standard term  "supergoal" is used for this concept in Eliezer Yudkowsky's earlier writings.
Contents
1 Terminal vs. instrumental vales
2 Human [...]s
3 Non-human [...]s
4 In a Friendly AI
5 Links
if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }
Terminal vs. instrumental vales
[...]s stand in contrast to instrumental values (also known as extrinsic values), which are means-to-an-end, mere tools in achieving [...]s. For example, if a given university student studies merely as a professional qualification, his [...] is getting a job, while getting good grades is an instrument to that end. If a (simple) chess program tries to maximize piece value three turns into the future, that is an instrumental value to its [...] of winning the game.
Some values may be called "terminal" merely in relation to an instrumental goal, yet themselves serve instrumentally towards a higher goal. However, in considering future artificial general intelligence, the phrase "[...]" is  generally used only for the top level of the goal hierarchy: the true ultimate goals of a system, those which do not serve any higher value.
Human [...]s
Humans' system of [...]s is quite complex. The values were forged by evolution in the ancestral environment to maximize inclusive genetic fitness. These values include survival, health, friendship, social status, love, joy, aesthetic pleasure, curiosity, and much more. Evolution's implicit goal is inclusive genetic fitness, but humans do not have inclusive genetic fitness as a goal. Rather, these  values, which were instrumental to inclusive genetic fitness, have become humans' [...]s (an example of subgoal stomp).
Humans cannot fully introspect their [...]s. Humans' [...]s are often mutually contradictory, inconsistent, and change over time.
Non-human [...]s
Future artificial general intelligences may have the maximization of a utility function or of a reward function (reinforcement learning) as their [...].  The function will likely be set by the AGI's designers.
Since people make tools instrumentally, to serve specific human values, the assigned value system of the artificial general intelligence may be much simpler than humans'. This will pose a danger, as an AI must seek to protect all human values if a positive human future is to be achieved. The paperclip maximizer is a thought experiment about an artificial general intelligence with consequences disastrous to humanity, with the the apparently innocuous [...] of maximizing the number of paperclips in its collection,
An intelligence can work towards any [...], not just human-like ones. AIXI is a mathematical formalism for modeling intelligence. It illustrates that the arbitrariness of [...]s may be optimized by an intelligence: AIXI  is provably more intelligent than any other agent for any computable reward function.
In a Friendly AI
For an artificial general intelligence to have a positive and not a negative effect on humanity, its [...] must be benevolent to humans. It must seek the maximization of the full set of human values (for the humans' benefit, not for itself).
Links
Eliezer Yudkowsky, Terminal Values and Instrumental Values
A: Terminal value

Q: The Craft and the Community
A: The final  sequence of Eliezer Yudkowsky's two-year-long string of daily posts to Less Wrong, on improving the art of rationality and building communities of rationalists.
A guide to this sequence is available at The Craft and the Community.

Q: The final  sequence of Eliezer Yudkowsky's two-year-long string of daily posts to Less Wrong, on improving the art of rationality and building communities of rationalists.
A guide to this sequence is available at [...].
A: The Craft and the Community

Q: The Fun Theory Sequence
A: A concrete theory of transhuman values.  How much fun is there in the universe; will we ever run out of fun; are we having fun yet; could we be having more fun.  Part of the complexity of value thesis.  Also forms part of the fully general answer to religious theodicy.
A guide to this sequence is available at The Fun Theory Sequence.

Q: A concrete theory of transhuman values.  How much fun is there in the universe; will we ever run out of fun; are we having fun yet; could we be having more fun.  Part of the complexity of value thesis.  Also forms part of the fully general answer to religious theodicy.
A guide to this sequence is available at [...].
A: The Fun Theory Sequence

Q: The Hanson-Yudkowsky AI-Foom Debate
A: In late 2008, an extensive and long-awaited debate about the Technological Singularity occurred on Overcoming Bias, mainly between Robin Hanson and Eliezer Yudkowsky. It focused on the likelihood of hard AI takeoff ("FOOM"), the need for a theory of Friendliness, and the future of AI, brain emulations, and recursive improvement in general. This debate is often used to illustrate the difficulty of resolving disagreements, even among expert rationalists.
The posts constituting this debate have been collected here to make it easier to follow for future audiences.

Q: In late 2008, an extensive and long-awaited debate about the Technological Singularity occurred on Overcoming Bias, mainly between Robin Hanson and Eliezer Yudkowsky. It focused on the likelihood of hard AI takeoff ("FOOM"), the need for a theory of Friendliness, and the future of AI, brain emulations, and recursive improvement in general. This debate is often used to illustrate the difficulty of resolving disagreements, even among expert rationalists.
The posts constituting this debate have been collected here to make it easier to follow for future audiences.
A: The Hanson-Yudkowsky AI-Foom Debate

Q: The Quantum Physics Sequence
A: A non-mysterious introduction to quantum mechanics, intended to be accessible to anyone who can grok algebra and complex numbers.  Cleaning up the old confusion about QM is used to introduce basic issues in rationality (such as the technical version of Occam's Razor), epistemology, reductionism, naturalism, and philosophy of science.  Not dispensable reading, even though the exact reasons for the digression are hard to explain in advance of reading.
A guide to this sequence is available at The Quantum Physics Sequence

Q: A non-mysterious introduction to quantum mechanics, intended to be accessible to anyone who can grok algebra and complex numbers.  Cleaning up the old confusion about QM is used to introduce basic issues in rationality (such as the technical version of Occam's Razor), epistemology, reductionism, naturalism, and philosophy of science.  Not dispensable reading, even though the exact reasons for the digression are hard to explain in advance of reading.
A guide to this sequence is available at [...]
A: The Quantum Physics Sequence

Q: The Science of Winning at Life
A: A sequence by lukeprog that summarizes scientifically-backed advice for "winning" at everyday life: in one's productivity, in one's relationships, in one's emotions, etc. Each post concludes with footnotes and a long list of references from the academic literature.

Q: A sequence by lukeprog that summarizes scientifically-backed advice for "winning" at everyday life: in one's productivity, in one's relationships, in one's emotions, etc. Each post concludes with footnotes and a long list of references from the academic literature.
A: The Science of Winning at Life

Q: The Sequences Summarized
A: An upcoming sequence that will summarize, for newcomers, much of the core content of The Sequences.

Q: An upcoming sequence that will summarize, for newcomers, much of the core content of The Sequences.
A: The Sequences Summarized

Q: The top 1% fallacy
A: Everyone thinks they're hiring the top 1%.  Martin Fowler said, “We are still working hard to hire only the very top fraction of software developers (the target is around the top 0.5 to 1%).”  I hear this from almost every software company. "We hire the top 1% or less," they all say.  Could they all be hiring the top 1%? Where are all the other 99%? General Motors?
When you get 200 resumes, and hire the best person, does that mean you're hiring the top 0.5%?  Think about what happens to the other 199 that you didn't hire.  They go look for another job.

Q: Everyone thinks they're hiring the top 1%.  Martin Fowler said, “We are still working hard to hire only the very top fraction of software developers (the target is around the top 0.5 to 1%).”  I hear this from almost every software company. "We hire the top 1% or less," they all say.  Could they all be hiring the top 1%? Where are all the other 99%? General Motors?
When you get 200 resumes, and hire the best person, does that mean you're hiring the top 0.5%?  Think about what happens to the other 199 that you didn't hire.  They go look for another job.
A: The top 1% fallacy

Q: The utility function is not up for grabs
A: The constraints of decision theory are only there to help you win; they don't specify what constitutes a win. Rationality in and of itself cannot constrain what you want, except insofar as what you thought you wanted failed to reflect what you actually wanted (or was just plain inconsistent). Hence the saying: the utility function is not up for grabs.

Q: The constraints of decision theory are only there to help you win; they don't specify what constitutes a win. Rationality in and of itself cannot constrain what you want, except insofar as what you thought you wanted failed to reflect what you actually wanted (or was just plain inconsistent). Hence the saying: [...].
A: The utility function is not up for grabs

Q: Third option
A: The phrase "tertium quid" (or "third option" in Latin) refers to something that exposes a dichotomy as false. A tertium quid is not somewhere between the two options (grey is not a tertium quid to the dichotomy of black vs. white), but rather something entirely new, something that escapes the one-dimensional line between two options.
—Tailsteak[1]
A third option is a way to break a false dilemma, showing that neither of the suggested solutions is a good idea.

Q: The phrase "tertium quid" (or "[...]" in Latin) refers to something that exposes a dichotomy as false. A tertium quid is not somewhere between the two options (grey is not a tertium quid to the dichotomy of black vs. white), but rather something entirely new, something that escapes the one-dimensional line between two options.
—Tailsteak[1]
A [...] is a way to break a false dilemma, showing that neither of the suggested solutions is a good idea.
A: Third option

Q: Timeless decision theory
A: A work-in-progress alternative to causal decision theory and evidential decision theory.

Q: A work-in-progress alternative to causal decision theory and evidential decision theory.
A: Timeless decision theory

Q: Topic that must not be named
A: When  Eliezer Yudkowsky established LW, he mentioned two topics that deserved a moratorium until the end of April 2009. These are The Singularity and Artificial General Intelligence. In discussions, these are often referred to as "The Topics that Must Not be Named". Occasionally you'll also see "The Institute that Must Not be Named." This is presumably SIAI (The Singularity Institute for Artificial Intelligence)

Q: When  Eliezer Yudkowsky established LW, he mentioned two topics that deserved a moratorium until the end of April 2009. These are The Singularity and Artificial General Intelligence. In discussions, these are often referred to as "The Topics that Must Not be Named". Occasionally you'll also see "The Institute that Must Not be Named." This is presumably SIAI (The Singularity Institute for Artificial Intelligence)
A: Topic that must not be named

Q: Traditional rationality
A: Traditional Rationality refers to the memes of rationality which are in general circulation. It is good enough to get Science done, but lacks the technical precision of Bayesian probability. Unsharpened by curiosity, it demands only the least effort needed to push forward human knowledge.
Traditional Rationality is phrased in terms of social rules, with violations interpretable as cheating - as defections from cooperative norms. If you want me to accept a belief from you, you are obligated to provide me with a certain amount of evidence. If you try to get out of it, we all know you're cheating on your obligation. A theory is obligated to make bold predictions for itself, not just steal predictions that other theories have labored to make. A theory is obligated to expose itself to falsification - if it tries to duck out, that's like trying to duck out of a fearsome initiation ritual; you must pay your dues.

Q: Traditional Rationality refers to the memes of rationality which are in general circulation. It is good enough to get Science done, but lacks the technical precision of Bayesian probability. Unsharpened by curiosity, it demands only the least effort needed to push forward human knowledge.
Traditional Rationality is phrased in terms of social rules, with violations interpretable as cheating - as defections from cooperative norms. If you want me to accept a belief from you, you are obligated to provide me with a certain amount of evidence. If you try to get out of it, we all know you're cheating on your obligation. A theory is obligated to make bold predictions for itself, not just steal predictions that other theories have labored to make. A theory is obligated to expose itself to falsification - if it tries to duck out, that's like trying to duck out of a fearsome initiation ritual; you must pay your dues.
A: Traditional rationality

Q: Transhumanism
A: Transhumanism is the position that it's ethical to radically improve the human condition using technology.

Q: [...] is the position that it's ethical to radically improve the human condition using technology.
A: Transhumanism

Q: Trivial inconvenience
A: Trivial inconveniences are inconveniences that take few resources to counteract but have a disproportionate impact on people deciding whether to take a course of action.

Q: [...]s are inconveniences that take few resources to counteract but have a disproportionate impact on people deciding whether to take a course of action.
A: Trivial inconvenience

Q: Truth
A: Correspondence between map and territory.
Alfred Tarski defined truth in terms of an infinite family of sentences such as:
"The sentence 'snow is white' is true if and only if snow is white."
To understand whether a belief is true, we need (only) to understand what possible states of the world would make it true or false, and then ask directly about the world.
'Truth' is a very simple concept, understood perfectly well by three-year-olds, but often made unnecessarily complicated by adults.

Q: Correspondence between map and territory.
Alfred Tarski defined [...] in terms of an infinite family of sentences such as:
"The sentence 'snow is white' is true if and only if snow is white."
To understand whether a belief is true, we need (only) to understand what possible states of the world would make it true or false, and then ask directly about the world.
'[...]' is a very simple concept, understood perfectly well by three-year-olds, but often made unnecessarily complicated by adults.
A: Truth

Q: Tsuyoku naritai
A: Tsuyoku naritai is Japanese.  Tsuyoku is "strong"; naru is "becoming" and the form naritai is "want to become".  Together it means "I want to become stronger" and it expresses a sentiment embodied more intensely in Japanese works than in any Western literature I've read.  You might say it when expressing your determination to become a professional Go player — or after you lose an important match, but you haven't given up — or after you win an important match, but you're not a ninth-dan player yet — or after you've become the greatest Go player of all time, but you still think you can do better.  That is tsuyoku naritai, the will to transcendence.

Q: [...] is Japanese.  Tsuyoku is "strong"; naru is "becoming" and the form naritai is "want to become".  Together it means "I want to become stronger" and it expresses a sentiment embodied more intensely in Japanese works than in any Western literature I've read.  You might say it when expressing your determination to become a professional Go player — or after you lose an important match, but you haven't given up — or after you win an important match, but you're not a ninth-dan player yet — or after you've become the greatest Go player of all time, but you still think you can do better.  That is [...], the will to transcendence.
A: Tsuyoku naritai

Q: Typical mind fallacy
A: The typical mind fallacy is the mistake of making biased and overconfident conclusions about other people's experience based on your own personal experience; the mistake of assuming that other people are more like you than they actually are.

Q: The [...] is the mistake of making biased and overconfident conclusions about other people's experience based on your own personal experience; the mistake of assuming that other people are more like you than they actually are.
A: Typical mind fallacy

Q: Ugh field
A: Pavlovian conditioning can cause humans to unconsciously flinch from even thinking about a serious personal problem they have. We call it an "ugh field". The ugh field forms a self-shadowing blind spot covering an area desperately in need of optimization.

Q: Pavlovian conditioning can cause humans to unconsciously flinch from even thinking about a serious personal problem they have. We call it an "[...]". The [...] forms a self-shadowing blind spot covering an area desperately in need of optimization.
A: Ugh field

Q: Underconfidence
A: Underconfidence is the state of being more uncertain than is justified, given your priors and the evidence you are aware of.

Q: [...] is the state of being more uncertain than is justified, given your priors and the evidence you are aware of.
A: Underconfidence

Q: Understanding
A: Understanding is more than just memorization of detached facts; it requires ability to see the implications across a variety of possible contexts.

Q: [...] is more than just memorization of detached facts; it requires ability to see the implications across a variety of possible contexts.
A: Understanding

Q: Unfriendly artificial intelligence
A: An Unfriendly artificial intelligence (or UFAI) is an artificial general intelligence capable of causing great harm to humanity, and having goals that make it useful for the AI to do so. The AI's goals don't need to be antagonistic to humanity's goals for it to be Unfriendly; there are strong reasons to expect that almost any powerful AGI not explicitly programmed to be benevolent to humans is lethal. A paperclip maximizer is often imagined as an illustrative example of an unFriendly AI indifferent to humanity. An AGI specifically designed to have a positive effect on humanity is called a Friendly AI.

Q: An [...] (or UFAI) is an artificial general intelligence capable of causing great harm to humanity, and having goals that make it useful for the AI to do so. The AI's goals don't need to be antagonistic to humanity's goals for it to be Unfriendly; there are strong reasons to expect that almost any powerful AGI not explicitly programmed to be benevolent to humans is lethal. A paperclip maximizer is often imagined as an illustrative example of an unFriendly AI indifferent to humanity. An AGI specifically designed to have a positive effect on humanity is called a Friendly AI.
A: Unfriendly artificial intelligence

Q: Universal law
A: Universal law is the idea that everything in reality always behaves according to the same uniform physical laws; there are no exceptions and no alternatives. The notion that there is such a thing as universal physical laws is one of the great, shocking discoveries of science. Just looking at the world from a human perspective, and knowing nothing of science, it is not at all obvious that underlying the chaotic world of tangible objects that we see, there should be a perfectly regular mathematically simple structure, and yet countless experiments have shown this to be the case.
Law is arguably a misleading term here: it's not as if things are naturally free to do "what they wish" and the laws of physics are some external constraint imposed from above by force or edict. Rather, what we know as "the laws of physics" are just a compact way of expressing how we think things really are at the lowest level. Our society's current conception of the laws of physics may be violated by some novel experiment under (what we would call) exotic conditions, but that only means we were mistaken about the true universal law, which can no more be "violated" than two and two can make five: it's just not how things work.

Q: [...] is the idea that everything in reality always behaves according to the same uniform physical laws; there are no exceptions and no alternatives. The notion that there is such a thing as universal physical laws is one of the great, shocking discoveries of science. Just looking at the world from a human perspective, and knowing nothing of science, it is not at all obvious that underlying the chaotic world of tangible objects that we see, there should be a perfectly regular mathematically simple structure, and yet countless experiments have shown this to be the case.
Law is arguably a misleading term here: it's not as if things are naturally free to do "what they wish" and the laws of physics are some external constraint imposed from above by force or edict. Rather, what we know as "the laws of physics" are just a compact way of expressing how we think things really are at the lowest level. Our society's current conception of the laws of physics may be violated by some novel experiment under (what we would call) exotic conditions, but that only means we were mistaken about the true [...], which can no more be "violated" than two and two can make five: it's just not how things work.
A: Universal law

Q: Unsupervised universe
A: A thought experiment developed to counter undue optimism - not just the sort due to explicit theology, but any sense that events and outcomes in our world are more destined, more proportionate, or more meaningful than we have a right to expect.
The thought experiment takes place in a simulated universe such as Conway's Game of Life, but with an overall benevolent God actually present to prevent sufficiently awful things from happening.  The thought experiment then consists of asking what would happen if we asked the mathematical question of what would happen according to the standard Life rules given certain initial conditions - so that even God cannot control the answer to the question; although, of course, God always intervenes in the actual Life universe.
The point of the thought experiment is to contrast our intuitions of a supervised universe and an unsupervised universe side-by-side - since the habits of thought associated with a "supervised universe" may carry over past explicit theology.
Example:  Would the twentieth century have gone differently, if Klara Pölzl and Alois Hitler had made love one hour earlier, and a different sperm fertilized the egg, on the night that Adolf Hitler was conceived?  For so many lives to turn on such a small event seems disproportionate, and a supervised universe would be run according to a plan in which things happened for sensible, proportionate reasons; such a small and insensible event would not have such a large influence on the sensible reasons of the plan.  It's possible to believe in a divine scheme of things without believing in God (e.g. Karl Marx) but such reasoning belongs to supervised universes rather than unsupervised ones.

Q: A thought experiment developed to counter undue optimism - not just the sort due to explicit theology, but any sense that events and outcomes in our world are more destined, more proportionate, or more meaningful than we have a right to expect.
The thought experiment takes place in a simulated universe such as Conway's Game of Life, but with an overall benevolent God actually present to prevent sufficiently awful things from happening.  The thought experiment then consists of asking what would happen if we asked the mathematical question of what would happen according to the standard Life rules given certain initial conditions - so that even God cannot control the answer to the question; although, of course, God always intervenes in the actual Life universe.
The point of the thought experiment is to contrast our intuitions of a supervised universe and an [...] side-by-side - since the habits of thought associated with a "supervised universe" may carry over past explicit theology.
Example:  Would the twentieth century have gone differently, if Klara Pölzl and Alois Hitler had made love one hour earlier, and a different sperm fertilized the egg, on the night that Adolf Hitler was conceived?  For so many lives to turn on such a small event seems disproportionate, and a supervised universe would be run according to a plan in which things happened for sensible, proportionate reasons; such a small and insensible event would not have such a large influence on the sensible reasons of the plan.  It's possible to believe in a divine scheme of things without believing in God (e.g. Karl Marx) but such reasoning belongs to supervised universes rather than unsupervised ones.
A: Unsupervised universe

Q: Utilitarianism
A: A moral philosophy that says that what matters is the sum of everyone's welfare, or the "greatest good for the greatest number".
Not to be confused with maximization of utility, or expected utility. If you're a utilitarian, you don't just sum over possible worlds; you sum over people.
Utilitarianism comes in different variants. For example, unlike standard total utilitarianism, average utilitarianism values the average utility among a group's members. Negative utilitarianism seeks only to minimize suffering, and is often discussed for its extreme implications.

Q: A moral philosophy that says that what matters is the sum of everyone's welfare, or the "greatest good for the greatest number".
Not to be confused with maximization of utility, or expected utility. If you're a utilitarian, you don't just sum over possible worlds; you sum over people.
[...] comes in different variants. For example, unlike standard total [...], average [...] values the average utility among a group's members. Negative [...] seeks only to minimize suffering, and is often discussed for its extreme implications.
A: Utilitarianism

Q: Utilitronium
A: Utilitronium is relatively homogeneous matter optimized for maximum utility (like computronium is optimized for maximum computing power). For a paperclip maximizer, utilitronium is paperclips. For more complex values, no homogeneous organization of matter will have optimal utility.
Utilitronium shockwave is a process of converting all matter in the universe into utilitronium as quickly as possible, which would look like a shockwave of utilitronium spreading outwards from the point of origin, presumably nearly at the speed of light.

Q: [...] is relatively homogeneous matter optimized for maximum utility (like computronium is optimized for maximum computing power). For a paperclip maximizer, [...] is paperclips. For more complex values, no homogeneous organization of matter will have optimal utility.
[...] shockwave is a process of converting all matter in the universe into [...] as quickly as possible, which would look like a shockwave of [...] spreading outwards from the point of origin, presumably nearly at the speed of light.
A: Utilitronium

Q: Utility
A: Utility is the amount by which preferences are satisfied. An abstract measure of utility is  called a util, or utilon. It is an arbitrary quantity, but can be consistently assigned once a set of utility functions are defined. All utils are equal; any diminishing returns or other factors are accounted for when utils are measured. Utils are not to be confused with hedons, units of subjective pleasure, or fuzzies, the warm feelings associated with the belief that one has accomplished good.
Utilitarianism is the philosophy that advocates whatever action produces the greatest amount of utility is the moral action.

Q: [...] is the amount by which preferences are satisfied. An abstract measure of [...] is  called a util, or utilon. It is an arbitrary quantity, but can be consistently assigned once a set of [...] functions are defined. All utils are equal; any diminishing returns or other factors are accounted for when utils are measured. Utils are not to be confused with hedons, units of subjective pleasure, or fuzzies, the warm feelings associated with the belief that one has accomplished good.
Utilitarianism is the philosophy that advocates whatever action produces the greatest amount of [...] is the moral action.
A: Utility

Q: Utility function
A: A utility function assigns numerical values ("utilities") to outcomes, in such a way that outcomes with higher utilities are always preferred to outcomes with lower utilities.
These do not work very well in practice for individual humans. Human drives are not coherent nor is there any reason to think they would be (Thou Art Godshatter), and even people with a strong interest in the concept have trouble working out what their utility function actually is even slightly (Post Your Utility Function). Furthermore, humans appear to calculate utility and disutility separately - adding one to the other does not predict their behaviour accurately. This makes humans highly exploitable.
pjeby posits humans' difficulty in understanding their own utility functions as the root of akrasia.
However, utility functions can be a useful model for dealing with humans in groups, e.g. in economics.

Q: A [...] assigns numerical values ("utilities") to outcomes, in such a way that outcomes with higher utilities are always preferred to outcomes with lower utilities.
These do not work very well in practice for individual humans. Human drives are not coherent nor is there any reason to think they would be (Thou Art Godshatter), and even people with a strong interest in the concept have trouble working out what their [...] actually is even slightly (Post Your Utility Function). Furthermore, humans appear to calculate utility and disutility separately - adding one to the other does not predict their behaviour accurately. This makes humans highly exploitable.
pjeby posits humans' difficulty in understanding their own [...]s as the root of akrasia.
However, [...]s can be a useful model for dealing with humans in groups, e.g. in economics.
A: Utility function

Q: Valid argument
A: An argument is valid when it contains no logical fallacies. Such arguments are not necessarily sound, because the premises may be false. For instance, the following syllogism is valid but has a false conclusion:

Q: An argument is valid when it contains no logical fallacies. Such arguments are not necessarily sound, because the premises may be false. For instance, the following syllogism is valid but has a false conclusion:
A: Valid argument

Q: Valley of bad rationality
A: Then I finally reply that my experience so far - even in this realm of merely human possibility - does seem to indicate that, once you sort yourself out a bit and you aren't doing quite so many other things wrong, striving for more rationality actually will make you better off. The long road leads out of the valley and higher than before, even in the human lands.
—Incremental Progress and the Valley
It has been observed that when someone is just starting to learn rationality, they appear to be worse off than they were before. Others, with more experience at rationality, claim that after you learn more about rationality, you will be better off than you were before you started. The period before this improvement is known as "the valley of bad rationality".

Q: Then I finally reply that my experience so far - even in this realm of merely human possibility - does seem to indicate that, once you sort yourself out a bit and you aren't doing quite so many other things wrong, striving for more rationality actually will make you better off. The long road leads out of the valley and higher than before, even in the human lands.
—Incremental Progress and the Valley
It has been observed that when someone is just starting to learn rationality, they appear to be worse off than they were before. Others, with more experience at rationality, claim that after you learn more about rationality, you will be better off than you were before you started. The period before this improvement is known as "the [...]".
A: Valley of bad rationality

Q: Wireheading
A: Wireheading is the direct stimulation of an individual's brain's reward center with electrical current. In both thought experiments and laboratory experiments, this makes the individual feel happy. In theory, wireheading with a powerful enough current would be the most pleasurable experience imaginable.
This raises difficult ethical questions for those who believe that morality is based on human happiness. A civilization of wireheads "blissing out" all day while being fed and maintained by robots would be a state of maximum happiness, but such a civilization would have no art, love, scientific discovery, or any of the other things humans find valuable. Anyone who would not value such a future for humanity, must consider a complexity of value stretching beyond simple happiness.
Wireheading is also an illustration of the complexities of creating a Friendly AI. Any AI naively programmed to increase human happiness could devote its energies to wireheading people, possibly without their consent, in preference to any other goals. Equivalent problems arise for any simple attempt to create AIs who care directly about human feelings ("love", "compassion", "excitement", etc).
There is some evidence that reward is distinct from pleasure, and that most currently hypothesized forms of wireheading just motivate a person to continue the wirehead experience, not to feel happy. However, there seems no reason to believe that a different form of wireheading which does create subjective pleasure could not be found.

Q: [...] is the direct stimulation of an individual's brain's reward center with electrical current. In both thought experiments and laboratory experiments, this makes the individual feel happy. In theory, [...] with a powerful enough current would be the most pleasurable experience imaginable.
This raises difficult ethical questions for those who believe that morality is based on human happiness. A civilization of wireheads "blissing out" all day while being fed and maintained by robots would be a state of maximum happiness, but such a civilization would have no art, love, scientific discovery, or any of the other things humans find valuable. Anyone who would not value such a future for humanity, must consider a complexity of value stretching beyond simple happiness.
[...] is also an illustration of the complexities of creating a Friendly AI. Any AI naively programmed to increase human happiness could devote its energies to [...] people, possibly without their consent, in preference to any other goals. Equivalent problems arise for any simple attempt to create AIs who care directly about human feelings ("love", "compassion", "excitement", etc).
There is some evidence that reward is distinct from pleasure, and that most currently hypothesized forms of [...] just motivate a person to continue the wirehead experience, not to feel happy. However, there seems no reason to believe that a different form of [...] which does create subjective pleasure could not be found.
A: Wireheading