Title: Superintelligence
Tags: superintelligence

Q: Associative value accretion
A: An approach to value-loading in which a mechanism is specified that leads the AI to adopt new final values over time and in a way that is contingent on its experiences. (Inspired by the way humans appear to acquire values.)

Q: Anthropic capture
A: A hypothesized phenomenon in which an AI thinks it might be in a simulation, and so tries to behave in ways that will be rewarded by its simulators.

Q: Augmentation
A: An approach to obtaining a superintelligence with desirable motivations that consists of starting with a system that already has appropriate motivations (e.g. a human being) and making it smarter, instead of attempting to engineer a goal system from scratch.

Q: Boxing methods
A: A control method that consists of engineering the AI's environment so as to reduce the AI's ability to interact with the outside world: e.g. running the AI in a carefully isolated virtual reality simulation and with only a restricted channel of communication to its human overseers.

Q: Capability control methods
A: Strategies for avoiding undersirable outcomes by limiting what an AI can do. Capability control methods include boxing methods, incentive methods, stunting, and tripwires.

Q: Cognitive enhancement
A: Improvements to a system's intellectual abilities.

Q: Collective superintelligence
A: A system composed of a large number of smaller intellects such that the system's overall performance across many very general domains vastly outstrips that of any current cognitive system.

Q: The common good principle
A: Superinelligence should be develpoed only for the benefit of all of humanity and in the service of widely shared ethical ideas.

Q: Crucial consideration
A: An idea or argument that might plausibly reveal the need for not just some minor course adjustment in our practical endeavors but a major change of direction or priority, for example by reversing the sign of the desirability of important interventions.

Q: Decisive strategic advantage
A: Strategic superiority (by technology or other means) sufficient to enable an agent to achieve complete world dominance.

Q: Direct specification
A: An approach to the control problem in which the programmers figure out what humans value, and then write code for the AI that explicitly contains the corresponding values or rules.

Q: Domesticity
A: An approach to the control problem in which the AI's motivation system is designed such as to give the AI very modest ambitions, limiting the range of things it wants to interfere with.

Q: Emulation modulation
A: Starting with brain emulations with approximately normal human motivations and modifying their motivations using digital drug analogs or other means.

Q: Evolutionary selection approach to value-loading
A: An approach to the value-loading problem that seeks to obtain an AI with desirable values through a process of iterated selection, analogous to the evolutionary selection process that produced humans.

Q: First principal-agent problem
A: The well-known problem faced by one human entitty ("the principal") that appoints another ("the agent") to act in the former's interest (found, for instance, in the relationship between employer and employee).

Q: Genie AI
A: An AI that carries out a high-level command, then waits for another.

Q: Humanity's cosmic endowment
A: The reservoir of physical resources in the universe that is accessible to a technologically mature Earth-originating civilization (starting from present-day Earth).

Q: Human-level AI
A: An AI that can match the intellectual performance of a typical human being in all practically important domains. (This term has significant ambiguities.)

Q: Human-level hardware
A: Hardware that matches the information-processing ability of the human brain.

Q: Human-level software
A: Software that matches the algorithmic efficiency of the human brain, for doing the tasks the human brain does.

Q: Impersonal perspective
A: The view that one should act in the best interests of everyone, including those who may be brought into existence by one's choices (cf. 'Person-affecting perspective').

Q: Incentive methods
A: Strategies for controlling an AI that consist of setting up the AI's environment such that the AI has instrumental reasons even if its final goals are not aligned with human values.

Q: Incentive wrapping
A: Provisions in the goals given to an AI that allocate extra rewards or influence to those who helped bring the AI about.

Q: Indirect normativity
A: An approach to the value-selection problem in which, instead of directly determining ourselves which values an AI is to promote, we specify a criterion or a method that the AI can follow, using its own intellectual resources to discover the concrete content of a merely implicitly defined normative standard.

Q: Instrumental convergence thesis
A: We can identify "convergent instrumental values", subgoals that are useful for the attainment of a wide range of possible final goals in a wide range of possible environments—subgoals that are therefore likely to be pursued by a broad class of intelligent agents.

Q: Intelligence explosion
A: A hypothesized event in which an AI rapidly improves from "relatively modest" to a radically superhuman level of intelligence (in a process that is often imagined to involve recursive self-improvement).

Q: Macrostructural development accelerator
A: An imaginary lever (used in thought experiments) that would alter the rate at which macro-structural features (such as technology and geopolitical dynamics) develop, while leaving unchanged the rate at which micro-level human affairs unfold.

Q: Mind crime
A: The mistreatment of morally relevant computational processes (in simulations, or inside an AI, or created in machine substrate for instrumental reasons).

Q: Moral rightness (MR) AI
A: An AI that seeks to do what is morally right.

Q: Motivational scaffolding
A: An approach to value-loading in which a seed AI is initially given simple goals that are later replaced with more complex (value-aligned) goals once the AI has developed sufficiently sophisticated representational resources.

Q: Multipolar outcome
A: A condition after the transition to the machine intelligence era in which there are multiple competing superintelligent agencies.

Q: Optimization power
A: The amount of quality-adjusted design effort that is being applied to improve a system's intelligence.

Q: Oracle AI
A: An AI that only answers questions.

Q: Orthogonality thesis
A: Intelligence and final goals are orthogonal: more or less any level of intelligence could in principle be combined with more or less any final goal.

Q: Person-affecting perspective
A: The view that one should act in the best interests of everyone who already exists or who will exist independently of one's choices (cf. 'Impersonal perspective').

Q: Perverse instantiation
A: An efficient way of satisfying an AI's goal that violates the intentions of the programmers who defined the goal criteria (e.g. achieving the goal 'make humans smile' by paralyzing human facial muscles into perpetual grins).

Q: Principle of differential technological developmnent
A: Retard the developmnent of dangerous and harmful techhnologies, especially ones that raise the level of existential risk; and accelerate the development of beneificial technologies, especially those that reduce the existential risks posed by nature or by other technologies.

Q: Principle of epistemic deference
A: A future superintelligence occupies an epistemically superior vantage point: its beliefs are (probably, on most topics) more likely than ours to be true. We should therefore defer to the superintelligence's opinion whenever feasible.

Q: Quality superintelligence
A: A system that is at least as fast as a human mind and vastly qualitatively smarter.

Q: Recalcitrance
A: How difficult a system is to improve.

Q: Recursive self-improvement
A: The process in which an AI (perhaps a seed AI) iteratively improves its own intelligence—using its increasing intelligence to apply increasingly strong optimization power to the task of cognitive self-enhancement.

Q: Reinforcement learning approach to the control problem
A: An approach in which the AI learns to maximize some notion of cumulative reward (where the reward signal is specified or administered by humans in some way intended to induce appropriate behavior on the part of the AI).

Q: Second principal-agent problem
A: The principal-agent problem confronting a human "principal" who wishes to design a superintelligent AI ("agent") to work on her behalf. Also referred to as "the control problem".

Q: Seed AI
A: An AI of initially modest capabilities that can boostrap into an impressive AI by improving its own architecture.

Q: Singleton
A: A world order in which there is, at the highest level of decision-making, only one effective agency (though it internally might contain many factions and interests); that is, a condition in which all important global coordination problems are mostly solved. Possible examples include a world democracy, an unchallenged world dictator, or a superintelligent AI powerful enough to suppress any potential rivals.

Q: Sovereign AI
A: An AI that acts autonomously in the world in pursuit of broad and possibly very long-range objectives.

Q: Speed superinelligence
A: A system that can do all that a human intellect can do, but much faster.

Q: State risk
A: A risk that comes from being in a certain state, such that the amount of risk is a direct function of the time spent there. For example, the state of not having the technology to defend against impacts carries risk proportional to the time we remain in that state.

Q: Step risk
A: A risk that comes from making a transition. Here the amount of risk is not a simple function of how long the transition takes. For example, traversing a minefield is not safer if done more quickly.

Q: Stunting
A: A control method that consist of limiting the AI's internal capabilities, such as by restricting the AI's access to information or engineering limitations to its cognitive faculties.

Q: Superintelligence
A: Any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest.

Q: Takeoff
A: The transition from a condition in which there is only human-level machine intelligence to one in which there is radical superintelligence. It is often characterized by its speed: a "Slow takeoff" takes decades or centuries, a "moderate takeoff" takes months r years, and a "fast takeoff" takes days or less.

Q: Technological completion conjecture
A: If scientific and technological development efforts do not effectively cease, then all important basic capabilities that could be obtained through some possible technology will be obtained.

Q: Technology coupling
A: A predictable timing relationship between two technoologies, such that develpoing one of the technologies has a robust tendency to lead to the development of the other, either as a necessary precursor or as an obvious and irresistible aplication or subsequent step. For example, whole brain emulation is plausibly coupled to neuromorphic AI, because a more primitive version of the technology that would be required for the former might already enable the creation of brain-inspired AI (and there would be strong incentives to use the technology to do that).

Q: Tool AI
A: An AI that is not "like an agent" but like a more flexible and capable version of contemporary software. In particular, it is not goal-directed.

Q: Value learning
A: An approach to the value-loading problem in which the AI learns the values that humans want it to pursue.

Q: Value-loading problem
A: The problem of causing the AI to pursue as final goals the values we want it to pursue.

Q: The wise-singleton sustainability threshold
A: A capability set exceeds the wise-singleton threshold if and only if a patient and existential rysk-savvy system with that capability set would, if it faced no intelligent opposition or competition, be able to colonize and re-engineer a large part of the accessible universe.

Q: Whole brain emulation
A: Machine intelligence created by copying the computational structure of the human brain.

Q: AI complete problem
A: A problem whose difficulty is essentially equivalent to the difficulty of building generally human-level intelligent machines.

Q: AI winter
A: A period of retrenchment, during which funding decreases and skepticism increases, and AI falls out of fashion.

Q: Ideal Bayesian agent
A: An agent that makes probabilistically optimal use of available information.

Q: Hedonium
A: Matter organized in a configuration that is optimal for the generation of pleasurable experience.

Q: Iterated embryo selection
A: A procedure that consists of genotyping and selecting a number of embryos that are higher in desired genetic characteristics; extracting stem cells from those embryos and converting them to sperm and ova; crossing the new sperm and ova to produce embryos; and repeating until large genetic changes have been accumulated.

Q: Infrastructure profusion
A: A phenomenon where an agent transforms large parts of the reachable universe into infrastructure in the service of some goal, with the side effect of preventing the realization of humanity's axiological potential.

Q: Riemann hypothesis catastrophe
A: An example of infrastructure profusion in which an AI, given the final goal of evaluating the riemann hypothesis, pursues this goal by transforming the Solar System into computronium—including the atoms in the bodies of whomever once cared about the answer.

Q: Computronium
A: Physical resources arranged in a way that is optimized for computation.

Q: Paperclip AI
A: An example of infrastructure profusion in which an AI, designed to manage production in a factory, is given the final goal of maximizing the manufacture of paperclips, and proceeds by converting first the Earth and then increasingly large chunks of the observable universe into paperclips.