The great thing, then, in all education, is to make our nervous system our ally instead of our enemy.… For this we must make automatic and habitual, as early as possible, as many useful actions as we can
For James, the idea of “habit” was defined at its core in terms of automaticity—that is, the degree to which we can perform an action automatically when the appropriate situation arises, without consciously entertaining the intention to do it
evolutionary history of humans, and while nearly all humans learn to understand and speak their native language with seemingly no effort, reading is a skill that takes years of education and practice to acquire. However, the skill of reading becomes automatic once it is acquired, in the sense that we can’t help but process the meaning of text that we see. The automatic nature of reading is seen in the well-known Stroop effect, in which a person is shown words written in colored ink and asked to name the ink color as quickly as possible. If we compare how long it takes to name the color of a word when the
skills are often very similar to habits in that they are executed automatically without any effort or awareness
The mindless nature of these routines is at odds with a long-standing idea in psychology that our actions are driven primarily by our goals and beliefs.2 However, research by psychologists Judith Ouellette and Wendy Wood has shown that many routine behaviors (especially those we engage in daily) are better explained in terms of how often they have been done in the past (that is, the strength of the habit) rather than in terms of goals or intentions
As we will see below, the idea that habits become detached from goals or intentions is one of the central concepts behind our knowledge of how habits work, and we have an increasingly deep understanding of how this comes to be.
The habits we have discussed so far all involve physical actions, but it’s important to point out that we can also have habits of mind.
habits of mind can become deeply disruptive, as when individuals suffering from obsessive-compulsive disorder become disabled by particular thoughts that they cannot keep out of mind
Finally, emotional responses to particular situations can also become habitual.
the psychological and physical responses that occur in a phobia can be thought of as an “emotional habit.”
While the gamut of habits thus spans from action to thought, most of the research on habits has focused on relatively simple actions
the human brain that there is much to be learned from studying them, though we always have to keep in mind that there are differences as well
One standard way that rodents are studied is to put them in an operant conditioning chamber (Figure 1.1), often called a “Skinner box” after the psychologist B. F. Skinner who popularized them for studying how rats learn
What Dickinson and his colleagues found was that early in the process of learning, the rats behaved as if they were goal directed: when the reward was devalued, the rats stopped pressing the lever. However, with additional training, the rats’ behavior became habitual, such that they continued to press the lever even though they didn’t want the reward. This transition from early reliance on goal-directed control to later reliance on habitual control is a pattern that we will see repeatedly in our examination of habits
Thus, habits differ from intentional, goal-directed behaviors in at least two ways: they are automatically engaged whenever the appropriate stimulus appears, and once triggered they are performed without regard to any specific goal. Now let’s ask why evolution would build a brain that is such a habit machine.
The computational neuroscientist Stephen Grossberg coined the term “stability-plasticity dilemma” to describe this conundrum: How does the brain known how to change at the right time without forgetting everything that it knows?
Our brains are thus stuck on the horns of a tricky dilemma. On the one hand, we would like for our brain to automate all the aspects of the world that are stable so that we don’t have to think about them
On the other hand, when things change in the world, we want our brain to remember those things
The basic strategy that evolution has used to solve the dilemma is to build multiple systems into the brain that support different types of learning. The psychologists David Sherry and Daniel Schacter proposed that these separate brain systems evolved because they were needed to solve a set of problems that are “functionally incompatible”—that is, problems that simply cannot be solved by a single system. They argued that the brain’s habit system evolved to learn those things that are stable (or invariant) in the world, whereas another memory system (known as the declarative memory system) evolved to allow us to learn the things that change from moment to moment
As we will see Chapter 8, many of the most effective ways of changing behavior involve changing the environment
Different scientists define habits in different ways, but most agree on a few basic characteristics. First, a habit is an action or thought that is triggered automatically by a particular stimulus or situation—it doesn’t require any conscious intention on our part. Second, a habit is not tied to any particular goal; rather, habits are engaged simply because of their trigger. This is important, because it means that the habit persists even if the reward that created it is no longer present. Third, habits are sticky: they come back despite our best efforts to suppress them, often when we are at our weakest point
dopamine plays a central role in strengthening actions that lead to reward, ultimately setting the stage for the development of habits.
our behavior arises from a competition between different learning systems in the brain
The complexity of the brain is beyond staggering, and what the general public may not know is that many neuroscientists quietly despair as to whether we can ever fully understand how it works
The signal first arrives in a structure buried deep in the brain called the thalamus, which can be thought
in via the thalamus
of as the brain’s switchboard—nearly all incoming signals to the brain come
striking feature of habits is how they can be completely divorced from our conscious memory for the past, both in their execution and in our later memory for them
Conscious recollection of the past relies specifically on the declarative memory system, which involves a set of brain areas in a deep part of the temporal lobe (known as the medial temporal lobe), including the hippocampus and the parts of the cerebral cortex that surround it
Damage to these regions can cause a loss of memory of the past as well as an inability to create new memories
One of the major findings of this work was that people with amnesia could learn not just new motor skills but new perceptual and cognitive skills as well
Their results showed that amnesic patients had no problem learning how to read the mirror-reversed text, improving their reading times just as fast as the healthy controls. When tested three months later, the amnesic patients also picked up right where they had left off, showing no loss of the skill, and in fact continued to improve on the task. These results provided a striking demonstration of just how much an individual can learn even in the face of amnesia, and they also provided clear evidence that the hippocampus and its related brain system are not necessary in order to learn new skills. But the question remained: If not the hippocampus, then what brain systems are essential for habits and skills?
The reptilian brain consists of a deep set of brain regions that are present in all vertebrate species, including the brain stem and the basal ganglia
MacLean highlighted the role of the reptilian brain in routine/habitual behavior, as well as in activities such as mating calls and displays of dominance or submission. The limbic system is a set of structures that MacLean thought were novel to mammals, which are involved in the experience of emotion. The neomammalian brain refers to the portion of the cerebral cortex that is most highly developed in mammals and that exploded in size as mammals evolved
When Martone and her colleagues tested the Huntington’s and amnesic patients on the mirror-reading task, they saw that the two groups exhibited almost the opposite pattern of deficits. The amnesic Korsakoff patients behaved very similarly to those in the study by Cohen and Squire, showing relatively normal learning of the mirror-reading skill but difficulty remembering the words that had appeared in the mirror-reading task. Conversely, the Huntington’s disease patients showed relatively normal ability to remember the words, and while they did benefit somewhat from practice on the mirror-reading task, their skill learning was substantially poorer than amnesics or controls. This established what we refer to as a double dissociation, in which two different groups show the opposite pattern of normal or impaired performance across two different tasks. This kind of dissociation is generally taken to provide good evidence that the different tasks rely on separate brain systems, and in this case the results provided some of the first evidence that people with basal ganglia disorders have impairments in skill learning
Interestingly, although MacLean’s ideas about the role of the basal ganglia in habitual behavior have stood the test of time, the idea that there is something particularly “reptilian” about these parts of the brain has been largely rejected by neuroscientists. Subsequent studies comparing the anatomy of brains of many different vertebrates (from reptiles to birds to mammals) have shown that the overall plan of brain organization is remarkably similar between these groups;8 even the lamprey, the most ancient living vertebrate, has a similar organization. Thus, the brain of a reptile is not fundamentally different from the brain of a human in its overall organization; the human simply has a lot more tissue, organized in a much more complex manner. As we will see in subsequent chapters, it is this development, particularly in the prefrontal cortex, that allows humans to go beyond the routine and habitual behavior that characterizes many other species such as lizards.
Deep within the brain sits a collection of brain nuclei (sets of cells bundled together) known as the basal ganglia. The basal ganglia in humans comprise several separate regions, including the caudate nucleus, putamen, and nucleus accumbens (which together are known as the striatum), the globus pallidus (which has two sections, internal and external), and the subthalamic nucleus, shown in Figure 2.3. In addition, the substantia nigra and ventral tegmental area, both of which include neurons that release dopamine, are considered part of the basal ganglia. While they are spread across different parts of the middle of the brain, what holds these areas together is the way in which they are tightly interconnected with each other
When the input from the cortex arrives at the striatum, it generally connects to a specific set of neurons known as medium spiny neurons because of their spiny appearance under a microscope. From here, there are two paths that the signals can take through the basal ganglia, which we refer to as the direct pathway and indirect pathway, both of which are shown in Figure 2.4. The direct pathway goes from the striatum to another region called the globus pallidus, specifically to the internal part of this region, while the indirect pathway takes a more circuitous route through the basal ganglia, as we will see later. From here, the signals are sent to the thalamus and are then sent back to the cerebral cortex, usually to a region that is very close to where the input originally initiated. It is for this reason that we refer to these circuits as corticostriatal loops.
are inhibitory, which means that when they fire they cause reduced activity in their target neurons in the globus pallidus. Those neurons in the globus pallidus are also inhibitory, such that when they fire they inhibit activity in their target neurons in the thalamus. And the neurons in the globus pallidus fire a lot—between 60 and 80 times per second when an animal is resting.9 This constant (or “tonic”) inhibition keeps the neurons in the thalamus largely silenced most of the time, preventing them from exciting their targets back in the cortex. Note what happened here: we have two inhibitory neurons in a row, which means that the input to the first one (the medium spiny neuron in the striatum) will reduce the constant inhibition of the second one (in the globus pallidus), leading to excitation in the thalamus and subsequently in the cortex. It’s like multiplying together two negative numbers, which makes a positive number. Thus, we think that the effect of stimulation of the direct pathway is usually to cause the initiation of an action or thought by exciting activity in the cortex at the end of the loop
How does the input from the cortex to the striatum know which pathway to take? It turns out that different groups of medium spiny neurons in the striatum send their outputs to either the direct or indirect pathway, and one of the main differences between those two sets of neurons has to do with everyone’s favorite neurochemical: dopamine.
First, let’s ask where dopamine comes from and what it does. The great majority of dopamine in the brain is produced in two small nuclei deep in the middle of the brain: the substantia nigra (specifically, a portion of this area called pars compacta) and the ventral tegmental area (see Figure 2.510). These neurons send projections to much of the brain, but the projections to the basal ganglia are especially strong. The number of dopamine neurons in the brain is tiny—about 600,000 in humans11—which belies their outsized effect on nearly every aspect of our thought and behavior. Dopamine is a neuromodulatory neurotransmitter, which means that it doesn’t directly cause excitation or inhibition in the neurons that it affects. Rather, it modulates the effect of other excitatory or inhibitory inputs to those neurons
One additional complication of dopamine (which also applies to the other neuromodulatory transmitters that we discuss later in the book, such as noradrenaline) is that there are different types of dopamine receptors that are present on neurons. Some of these (known as D1-type receptors) have the effect of increasing the excitability of the neurons where they are present (turning up the volume), while others (D2-type receptors) have the effect of reducing the excitability of those neurons (turning down the volume)
The loss of these neurons starves the brain of dopamine, which results in a relative increase in activity in the indirect pathway, since dopamine suppresses activity on those neurons due to their D2-like receptors. Conversely, the lack of dopamine results in a decrease in activity in the direct pathway, since dopamine increases the activity of those
neurons due to their D1-like receptors
As I mentioned above, dopamine has many different effects in the brain, and one of these is central to its role in the formation of habits: it modulates the basic mechanism of change in the brain, known as synaptic plasticity
Synaptic plasticity is the process by which experience changes the strength of synapses, so that some neurons become more potent at exciting other neurons and others become less potent. This plasticity is thought to be critical to learning
One of the most common forms of plasticity happens when one cell causes another to fire in quick succession and the strength of their synapse is increased. This kind of plasticity (known as Hebbian plasticity after the neuroscientist Donald Hebb) is often described in the following terms: “Cells that fire together, wire together.” In some regions of the brain, including the striatum, this concept is slightly modified to get the three-factor rule: “Cells that fire together, in the presence of dopamine, wire together; cells that fire together without dopamine come unwired.”
In the beginning of the experiment, before the monkey knew that the light was predictive of the reward, the dopamine neurons didn’t fire until the reward appeared. But once the monkey learned that the reward was foreshadowed by the light, the dopamine neurons fired when the light appeared and did not fire when the reward appeared. Further, if the expected reward did not appear after the light, then activity in the dopamine neurons went down below their baseline level of activity. This was the first hint that dopamine neurons are not strictly sensitive to reward, but instead seem to be sensitive to situations where the world is different from our predictions (a concept known as reward prediction error)
This discovery was critical because it helped link dopamine with a set of ideas from computer science and psychology that ultimately led to what is now the dominant computational framework for understanding the role of dopamine. Within computer science, researchers have long been interested in how to build systems that can learn from experience; this field is now known as machine learning and is the foundation for many of the automated systems we interact with every day. One of the kinds of learning that these researchers have investigated is called reinforcement learning, which basically means learning by trial and error
One of the basic ideas from the theory of reinforcement learning is that learning should proceed on the basis of how well our predictions match the outcomes that we actually experience. After all, if we can perfectly predict the world then there is nothing else to learn! Most theories of reinforcement learning posit that the decision maker chooses an action based on the predicted value of all the possible actions one could take; in the case of our two slot machines, this would mean choosing the machine with the highest predicted value. We then observe the outcome (win or loss), and use this information to update our prediction for the next round. Importantly, it’s not the absolute amount of the win or loss that we use to update our predictions—rather, it’s the difference between the prediction and the observed outcome that we use, which is exactly the prediction error signal that dopamine was shown by Schultz and his colleagues to represent. By showing that dopamine could be understood in terms of the mathematical theory of reinforcement learning, this work provided a powerful framework that continues to be highly influential in the study of decision making in the brain
a major discovery over the last two decades is that dopamine is not directly responsible for the pleasurable sensations that occur due to drug use. Instead, the role of dopamine appears to be centered on motivation—or as the neuroscientist Kent Berridge has called it, “wanting,” rather than “liking.”
that the role of dopamine is what they call “incentive salience”: rather than determining how much an organism likes a reward, dopamine instead provides signals about how much the organism wants some particular reward in the world and how much they will work to obtain it
It is dopamine receptors in the nucleus accumbens (a portion of the striatum that is heavily connected to other parts of the brain involved in emotion) that appear to play a central role in incentive motivation, but its role is complex. Blocking dopamine in the nucleus accumbens doesn’t seem to interfere with the basic appetite for food or with the pleasure that is obtained by eating it. However, it does interfere with the animal’s willingness to engage in behaviors required to obtain food or to work extra for additional food
While blocking dopamine did not reduce the animals’ expressions of hedonic responses, blocking of opioid neurotransmitters did. These results are consistent with numerous reports on the effects of naltrexone, a drug that blocks opioid transmission and is commonly used for the treatment of alcoholism. Studies have examined the effects of naltrexone on everything from sex to gambling to amphetamine administration, and have generally found that the drug reduces the pleasure that individuals experience from each of these.
The research in this chapter has shown us that the basal ganglia are the center for habit learning in the brain, and that dopamine plays a critical role in establishing new habits
To understand how someone could go for so long only to relapse into addiction, we need to understand the neuroscience behind why habits are so sticky.
We learned in Chapter 1 that the brain has to determine when to remain stable and when to change, known as the stability-plasticity dilemma
Bouton’s work on resurgence and related phenomena has helped to cement the idea that when we supplant an old behavior with a new one, we are not actually forgetting the old habit—instead, we are actively inhibiting the original behavior so that the new behavior can emerge. What he has further shown is that this inhibitory learning is much more closely tied to the context in which it is learned than is the original habit, and this idea has important consequences for the treatment of many different disorders that involve habitual thoughts or actions, from phobias to post-traumatic stress disorder to obsessive-compulsive disorder
While both groups showed much less fear of the spider a week after exposure than they had prior to treatment, Craske and her colleagues found that changing the context reduced the effectiveness of the therapy—the students who underwent the follow-up in the same room as the initial treatment showed less fear than those who saw the spider again in a different context.
To understand this, we first need to dive more deeply into the structure of the basal ganglia. Remember from Chapter 2 that the striatum is connected to the cerebral cortex by a circuit known as a corticostriatal loop. There are two aspects of the anatomy of these loops that are important for understanding how habits develop. First, these loops are fairly specific in their anatomy, such that the outputs of the loop return to roughly the same place in the cerebral cortex from which the inputs to the striatum arose. Second, different parts of the basal ganglia receive inputs from different parts of the frontal lobe, but these inputs are not random: specific regions in the cortex send their connections to specific regions in the basal ganglia
Yin and Knowlton subsequently developed a framework for understanding how habit learning proceeds in the brain.2 It starts with goal-directed learning that initially involves the “cognitive“ corticostriatal loop that connects the dorsolateral prefrontal
cortex and caudate nucleus. Over time the “motor” circuit involving the motor cortex and putamen starts to learn the habit, and ultimately takes over from the cognitive loop.
This means that the striatal neurons that are part of the cognitive loop also send some input to the dopamine cells that ultimately send their outputs to the motor loop. Yin and Knowlton proposed that this feature of the dopamine system allows the cognitive system to slowly ingrain a motor habit by sending dopamine signals and thus modulating plasticity in the motor loop. As the habit becomes ingrained in the motor system, it becomes less amenable to inspection by the cognitive systems, leading to behaviors that we can become completely unaware of
Anne Graybiel is a neuroscientist at MIT whose work has provided detailed insight into how our brain chunks actions as we acquire a new habit. Using all of the powerful neuroscientific tools we have described so far, her work has shown that as rats develop a new habit, activity in the basal ganglia “bookends” the sequence of actions that comprise the habit, such that once the sequence begins it can run to completion without additional activity
Graybiel’s early research showed that when rats are first learning this task, there is activity in the striatum throughout their run; but as the turn toward the reward becomes a habit, the activity occurs primarily at the beginning and end of the action
As the rats developed the habit, the “action-bracketing” pattern (in which activity occurs only at the beginning and end of the run) began to develop in the striatum, and as this happened the rats began to deliberate less
This result shows that, as habits develop, the striatum and prefrontal cortex work together to transform the action sequence into a single unit of action rather than a set of individual actions, making it much harder to stop in the middle of the sequence once it is triggered.
Simply seeing another person smoking or walking into a smoky bar can cause a smoker to have an immediate urge to light up a cigarette
he injected some of the rats with lithium chloride, which makes them lose their appetite. Rats who had only received
a bit of training on the lever-pressing response were less likely to press the lever after being injected with the toxin; this means that their behavior was goal directed, since the feeling of sickness reduced the attractiveness of the value of food. On the other hand, rats who had received many hours of training on the lever press continued to press just as much as the rats who were not injected with the toxin, which is a hallmark of habitual behavior. When he examined the amount of Pavlovian-instrumental transfer, he saw that it was directly related to whether the response was habitual or not; rats with little training showed little transfer, whereas rats with many hours of training showed a large amount of transfer, and it was unaffected by devaluation of the food. That is, once the behavior becomes a habit, it can be triggered by a related stimulus even though the animal doesn’t actually want the reward anymore!
When they specifically interfered with dopamine signaling in the nucleus accumbens, they found that the Pavlovian-instrumental transfer was reduced, whereas interfering with dopamine signaling in a different part of the brain (the middle part of the rat’s prefrontal cortex) did not affect transfer. This shows that dopamine in the nucleus accumbens plays a central role in tying trigger cues to reward-seeking behaviors.
In between reflexes and goal-directed behaviors lie habits. Habits are actions that at some point may have been goal directed, but with enough repetition become automatic, very much like a reflex—except that while reflexes are basically impossible to stop, habits can often be stopped with sufficient effort and attention.
What Packard found was that the rats’ behavior differed depending on how much training they had received. Early in training, the rats exhibited the hallmark of place learning: when put in the opposite arm, they made the correct turn to get to the previous location of the food. However, with more training the behavior changed, with the rats now making the response they had learned before. Thus, habits developed with experience, just as they had in Dickinson’s original experiments
Packard then made an important mental leap. Previous work on memory systems had nearly always assumed that the different systems worked independently of one another, but Packard proposed that both systems were always learning but they then competed to determine how the animal would behave. This would imply that disrupting the brain system involved in one kind of learning should shift the animal to using the other kind of learning. On the other hand, if the other system was not involved, then the rat should simply start exploring the two arms randomly, since it wouldn’t have any way to know which arm had food. The results showed that the systems did indeed seem to be both learning at the same time and competing with each other to control the rat’s behavior: when he deactivated the
response learning strategy, whereas when he disrupted the basal ganglia later in learning, the rats shifted to a place learning strategy. He also showed that he could shift animals toward using one kind of learning or the other by chemically stimulating the brain area that supports that kind of learning. This work suggested that the brain’s different memory systems were in constant competition to determine how we behave.
We used functional MRI to measure brain activity in the basal ganglia and medial temporal lobe while individuals performed either the trial-and-error or paired-associate version of the weather prediction task. When we compared brain activity across these two different versions of the task, we found results that were consistent with our understanding of the brain’s memory systems: there was greater activity in the basal ganglia for the trial-and-error version of the task, whereas there was greater activity in the medial temporal lobe for the paired-associate version of the task. We also found something that led us to agree with Packard that these two systems are in competition with one another: their activity seemed to move in opposite directions
The quest for artificial intelligence began in the 1950s, and for many decades largely focused on developing systems that reasoned like humans on difficult tasks, such as medical diagnosis or chess. These approaches languished, unable to even start to solve human-level problems in a robust and flexible way. But in the twenty-first century a different approach to artificial intelligence has shown itself to be much more adept at solving the kinds of problems that are necessary to acheive human-level intelligence. These methods, which go by the name machine learning, take advantage of very powerful computers along with large amounts of data in order to learn in a way a bit closer to how humans learn. In particular, an approach known as deep learning has been highly successful at solving a number of problems that vexed computer scientists for many years
In between these two types of learning sits reinforcement learning, which we have already heard about in the context of dopamine. In reinforcement learning, the system must learn the appropriate actions based on feedback from the world, but it isn’t told explicitly what the right answer is—it simply receives carrots or sticks depending on whether it makes the right choice
in 1968 the psychologist Leon Kamin showed that the association between a stimulus and a reward could be blocked if the reward was already associated with another stimulus. For example, in grade school we might come to associate the sound of a particular bell with lunchtime, such that the occurrence of the bell would make us start to salivate. What Kamin showed was that if another stimulus was later added—for example, a flashing light along with the bell—the relationship between that second stimulus and the outcome was blocked, such that if the light occurred later on its own, it would not elicit the same response as the bell. This showed that the brain was not simply recording which stimuli went together in the world. Rescorla and Wagner developed a mathematical theory of learning based on the idea that learning depends on the degree to which their predictions in the world were violated—exactly the same idea of reward prediction error that we encountered in our discussion of dopamine. While this particular theory has largely been superseded by newer approaches, it cemented the concept of error-driven learning in psychology
The most basic model has several major components. The policy describes how actions are chosen in any particular state
we usually want a policy that allows some degree of exploration, such that we can occasionally pick a machine that we don’t currently think is very good just to make sure that we are right about that. The model also needs a reward signal, which tells it the outcome of its action. In our case, this is simple—we just record whether we won or not for each trial
Suppose that we randomly choose machine 2 on the first trial, and we happen to win $1 (which for that machine happens 40% of the time). The next job of the model is to update its value estimates based on that experience—actually, based on how our experience differs from our expectation
In this case, the value that we expected for machine 2 on the first trial was 1; the reward prediction error is thus 1. We update the value estimate for machine 2 by adding the reward prediction error to the existing value estimate, but only after multiplying it by a relatively small number (called a learning rate) to make sure that any particular win doesn’t have a huge impact on our value estimates—this helps prevent our behavior from changing too quickly based on limited evidence, stabilizing our behavior over time
Why is a mathematical model of learning relevant for our understanding of habits? Remember our discussion in Chapter 2 of the work of Wolfram Schultz, who studied how dopamine cells in the monkey’s brain responded to rewards and the cues that predicted them. His research showed that the firing of dopamine neurons very closely matched the difference between actual outcomes and predicted outcomes—exactly the prediction error that is computed in the reinforcement learning model. Subsequent research by Hannah Bayer and Paul Glimcher made this link even tighter, showing a strong mathematical relationship between the activity of dopamine neurons in the monkey’s brain and the prediction error values obtained from a reinforcement learning model.4 This is an example of what is now a burgeoning research area, known as computational neuroscience, in which models from computer science are used to understand how brains work.
The model-free learner sounds remarkably silly, but it turns out that it actually seems to provide a good description of how habits work, in the sense that it simply performs the learned response given the situation, without regard to goals or other knowledge. Another kind of reinforcement learning system, known as model-based reinforcement learning, uses structured knowledge to understand how the world works and make decisions accordingly. When we think of a “model” of the world, we often use the concept of a map. This could be a map of a physical space (like a road map), but it could also be some other kind of “cognitive map” that outlines our knowledge of the world
There is also evidence that specific situational factors can affect the deployment of model-based versus model-free reinforcement learning. In particular, distraction seems to drive people toward the use of model-free control.
Otto and his colleagues presented subjects with the two-step decision-making task, which they performed under either focused or dual-task conditions. What he found was that whereas subjects behaved as model-based learners when they were focused, they were more likely to use model-free learning when they were distracted. There is a large body of research that implicates the prefrontal cortex in multitasking, and Otto’s results are consistent with the idea that the prefrontal cortex is necessary for model-based decision making, such that engaging it in multitasking reduces its effectiveness and allows the model-free system to win the competition.
While most research has focused on habitual actions, there is increasing interest in the idea that goals can become habitual as well.
Part of the reason that psychologists focus so much on self-control is that it seems to have powerful effects on many important life outcomes. Some of the most compelling evidence for this has come from research by Terrie Moffitt and Avshalom Caspi from Duke University, who have spent years following a group of more than 1000 individuals born 1972–1973 in Dunedin, New Zealand. They first measured self-control when these individuals were children, starting at 3 years old, by simply asking parents, teachers, and the children themselves whether they showed evidence of self-control problems, such as acting before thinking, difficulty waiting or taking turns, tendency to “fly off the handle,” and low tolerance for frustration. In a set of studies, Moffitt and Caspi have examined how these early measures of self-control relate to social, educational, and health outcomes in adulthood. The results are striking, to say the least: Nearly every positive life outcome is better for the children who had higher self-control at a young age. They are more likely to be financially successful, have better physical health, are less likely to have drug and alcohol problems, and are less likely to be convicted of a crime, just to name a few of the outcomes. Perhaps most importantly, higher self-control appeared to help these individuals avoid what Moffitt and Caspi called “snares,” or life choices that end up trapping individuals into undesirable outcomes—such as starting to smoke at an early age or dropping out of school.
A study published in 2018 examined 194 individuals from this registry to assess the kinds of personality changes that result from brain damage.4 Looking at individuals with damage across the entire brain, they found that almost half of these individuals showed some kind of change in their personality, which took several different forms. The most common effect was social and emotional dysfunction, similar to what was seen in Phineas Gage. Another common effect was “executive dysregulation,” involving a range of symptoms including lack of judgment, indecisiveness, and social inappropriateness. Another group showed symptoms similar to those observed in Rosemary Kennedy’s case, involving apathy, social withdrawal, and a lack of stamina or energy. Finally, some individuals showed signs of emotional distress and anxiety.
Somewhat surprisingly, they found that more than half of the patients had some aspect of their personality that was rated as improved after their brain damage compared to before, and that these individuals were more likely to have damage to the farthest forward (or anterior) parts of the frontal lobe. Some of the examples provided in the paper show just what kind of changes occurred. One patient had been highly irritable and outspoken prior to undergoing surgery for a tumor in her frontal lobe, and was described as “stern” by her husband. After the surgery, she became much happier and outgoing, and her husband noted that she smiled and laughed more. Another patient had been frustrated and angry prior to a brain aneurysm, often complaning about his job and being temperamental with his daughter; he was also described by his wife as being “mopey.” After the aneurysm, which caused damage to part of his prefrontal cortex, he became much more easygoing and content, and both he and his wife described the changes in his personality as being positive. These findings show that some of the deepest aspects of our personalities live in the far reaches of the frontal lobes, even the not-so-good aspects
If we ask what makes the prefrontal cortex so important for self-control, the key feature is its wiring
These regions are arranged in a hierarchy, with the prefrontal cortex at the very top, receiving input from each of the lower-level unimodal cortical regions. In addition, even within the prefrontal cortex there is a hierarchy, with regions toward the front processing more complex information. In this way, the regions at the top of the hierarchy (which sit at the very front of the prefrontal cortex) have access to an “executive summary” of all the brain’s available information
The ways in which these differences in the architecture of tissue give rise to huge differences in cognitive ability between species are yet to be fully understood
It was once thought that the human prefrontal cortex was outsized compared to other primates, but recent evidence using MRI scanning of the brains of many different primate species (monkeys, great apes, and humans) has shown that the size of the human prefrontal cortex relative to our brain size is at least roughly similar to that of other apes
The prefrontal cortex is also the last part of the brain to develop (which will be of no surprise at all to anyone who has spent much time around adolescent children)
and synapses is followed by a prolonged period of “pruning” in which unnecessary neurons and connections are removed. This growth and subsequent pruning happens earliest in the primary sensory and motor regions, finishing within a few years after birth, whereas in the prefrontal cortex the pruning doesn’t really kick in until mid-childhood and isn’t complete until early adulthood
The process of myelination begins in utero and continues throughout childhood, but in the prefrontal cortex it takes much longer, extending into early adulthood
The protracted development of white matter in adulthood was especially apparent in white matter regions connecting the prefrontal cortex to the rest of the brain. These results, along with many others, show that white matter continues to develop long after the cortex has reached its adult zenith.
The discovery that diffusion-weighted imaging could be used to image the structure of white matter led to the development of a number of techniques for measuring the structure of white matter, the most notable of which is called diffusion tensor imaging. This technique involves the collection of diffusion-weighted images along six different directions, which allows us to fit a mathematical model to the data that quantifies the direction and shape of diffusion at each point in the brain. In particular, we can compute a measure known as fractional anisotropy that quantifies the degree to which the diffusion is isotropic or anisotropic. While this is not a pure measure of myelination, it is related, and it has allowed researchers to gain a much better handle on how the structure of white matter relates to many different aspects of brain function and development
Goldman-Rakic and her colleagues also established that dopamine was critical for working memory. In one study, they administered a drug directly into the prefrontal cortex that blocked the function of dopamine D1 receptors in monkeys.9 As the drug took effect, the monkeys made many more errors in remembering the intended eye movements, and the errors became greater as the delay between the cue and the response became longer, suggesting that the memory was being degraded over time. In a later study, they applied the dopamine-blocking drug while also recording from neurons in the prefrontal cortex.10 While the drug had minimal effects on neurons that were responding to the cue or the movement, it resulted in decreased activity in the cells that responded during the delay
as Miller and his colleagues said in a 2018 overview of their research, “in the constant chatter of the brain, a brief scream is heard better than a constant whisper.”11
noradrenaline is just a small chemical change away from that molecule. Noradrenaline is also chemically very similar to dopamine and in fact is created directly from dopamine in the brain; both are members of a class of neurochemicals called catecholamines. This conversion happens in a very small region of the brain known as the locus coeruleus, buried deep in the brain stem; its name is Latin for “blue spot,” due to the fact that the region appears blue when viewed in a dissected brain. Just like the dopamine system, the locus coeruleus sends its projections widely across the brain, especially to the prefrontal cortex. And just like dopamine it also appears to play a central role in working memory.
Many people take a type of drug called a “beta-blocker” for high blood pressure, which gets its name from the fact that it blocks a specific version of noradrenergic receptor known as the beta receptor; this also highlights the fact that chemicals like noradrenaline and dopamine play many roles across our entire body, not just in our brain. Beta receptors are responsible for the usual effects of adrenaline that we think of; in fact, many people (including myself at one time) take beta-blockers when they have to speak in public, because they reduce some of the symptoms of anxiety
Arnsten has argued that there is an “inverted-U” relationship between the level of catecholamines in the prefrontal cortex and the function of the neurons there (see Figure 5.414). Just as in the classic story of Goldilocks and the three bears, the level of catecholamines in the prefrontal cortex needs to be “just right” for optimal function; if the level is too low (as it is thought to be when we are sleepy) or too high (as occurs when we are under extreme stress), the prefrontal cortex becomes unreliable and our ability to think and plan goes out the window
Examination of cognitive performance after the first three days of Hell Week showed that the soldiers were badly impaired on tests of memory and attention, compared to their performance just before the week started. For example, on a task that required them to learn a sequence of keystrokes on a computer keypad, the solders took more than twice as long to learn the task after three days of intense stress
When we are alert and interested, moderate amounts of noradrenaline are released into the prefrontal cortex, which optimizes the function of neurons by making their patterns of firing more precise. These moderate levels of noradrenaline engage a particular group of noradrenaline receptors (the alpha-2A receptors mentioned above), which strengthens connectivity between neurons in the prefrontal cortex, allowing them to better hold onto information over time. Research by Arnsten and her colleagues showed that applying the drug guanfacine (which activates alpha-2A receptors) directly to the prefrontal cortex in monkeys during a working memory task caused neurons in the area to fire more precisely when a low level of the drug was applied, while high doses of the drug disrupted the ability of neurons to fire during a delay period
Mischel readily admitted that the sample size of 35 children was far too small to make any strong conclusions, but as is often the case, the subsequent discussions of the results dropped this important caveat, and it soon became common parlance that this study had demonstrated that the ability to delay gratification was an essential component of success, right alongside intelligence
In a first study, Angela Duckworth and her colleagues examined how the ability to delay gratification in this dataset was related to a number of outcomes when the children were in eighth or ninth grade.17 They found that waiting on the delay of gratification test at 4 years of age was related to higher GPA and standardized test scores in adolescence. Interestingly, they also found that waiting was related to body mass index; children who had trouble waiting were more likely to be overweight. They also tested whether the relationship between self-control and academic outcomes was caused by greater intelligence in children who waited longer, using an additional set of self-control and intelligence measures that had been collected on these children. They found that the relationship between waiting and grade point average was primarily driven by self-control and not by intelligence
The differences in self-control between the children in these two groups were striking. The children from degreed mothers waited on average about 90 seconds more than the children with nondegreed mothers and were also less than half as likely to ring the bell within the first 20 seconds; they also
he found that the relationship between waiting and academic performance in these children was primarily present for kids who rang the bell in the first 20 seconds; for kids who could wait longer than that, there didn’t seem to be a relation between how much longer they waited and how well they did academically.
Other research has also shown that children are much less likely to wait when they don’t trust the experimenter, or when they don’t trust other people in general. These findings show that it can be very difficult to disentangle the different factors that may lead to relationships between delay of gratification and academic performance, but the findings nonetheless provide a good basis for thinking that the ability to delay gratification is a reliable correlate of success later in life.
When people make these kinds of choices, they tend to overweight immediate rewards more strongly than economic theory says they should. One consequence of this is that their choices are “dynamically inconsistent,” meaning that their relative preference for different outcomes changes over time. Let’s say that today I offer you a choice between 30 in four weeks. Nearly everyone will choose the larger/later reward. However, now fast-forward two weeks, such that the same choice becomes 30 in two weeks. In this case, many people will switch their preference and take the immediate reward. This suggests that people will differently value the same outcomes depending on when they are considering them, violating the basic rules of classical economics
When we look across people, we see that k differs widely. For example, one large study measured k in more than 20,000 people and found that the highest k value across these people was more than 1000 times larger than the lowest value! In that study, the most patient person would prefer to wait 30 days for 20, whereas the most impatient person would require a delayed reward of 20. These differences between people in their discounting rates appear to arise from a combination of genetic and environmental influences. We already saw some of the environmental influences above, when we discussed the marshmallow task. If a person doesn’t trust others, then they are more likely to take the immediate reward rather than waiting for a delayed reward that they don’t trust others to actually deliver. Another factor that likely impacts discounting rates is one’s socioeconomic status. The behavioral economists Sendhil Mullainathan and Eldar Shafir have proposed that when someone experiences scarcity (as the poor do on a daily basis), their attention is so focused on solving their immediate problems that thinking about the future just doesn’t make sense. This could be why poor individuals take out high-interest payday loans; as Mullainathan and Shafir showed in a set of studies, scarcity (in this case, in a video game) causes people to focus more on immediate needs and thus be more likely to borrow against their future.19 Consistent with this idea, research has also shown that income has a direct relationship with discounting rates, such that lower-income individuals show faster discounting of delayed monetary rewards compared to higher-income individuals
Just as behavior on the marshmallow task is related to life outcomes, so is discounting of future monetary rewards. This has been shown most clearly for drug abuse, where a large number of studies have confirmed that individuals with drug addictions show substantially faster discounting than do nonaddicted individuals
Within both neuroscience and economics, the idea of a battle between impulsive and rational brain systems has played a major role in explanations of intertemporal choice, in the form of dual systems theories of decision making. These theories propose that there are two brain systems that play a role in decision making. One system, referred to as the “doer” by the economists Thaler and Shefrin,26 the “hot system” by Walter Mischel, and “System 1” by Daniel Kahneman, is an automatic system that drives us toward fast and immediate consumption of rewards without regard to goals. This system is usually associated with brain regions that are engaged by reward, including the nucleus accumbens, the ventromedial prefrontal cortex, and the dopamine system. A second system (referred to variably as the “planner,” “cold system,” or “System 2”) is thought to be a rational, goal-directed, and patient thinker. This system is generally linked to the lateral parts of the prefrontal cortex, which have long been associated with what neuroscientists call cognitive control processes. These processes, which include holding information in working memory, resisting distraction, planning future actions, and inhibiting unwanted actions, are thought to be the basic ingredients of self-control.
More impulsive individuals had lower household income and education levels, higher body weight, and a greater likelihood of having experimented with drugs. The availability of the genetic data also allowed the researchers to examine what is called the genetic correlation between these different traits—that is, to what degree is similarity in these traits related to similarity in the genomes of different individuals? Intriguingly, this showed that there were strong genetic correlations between impulsivity and a number of negative outcomes, including drug usage and mental health problems like depression and ADHD. This tells us that the genetic risk for all of these negative outcomes is related (at least in part) to impulsivity.
my Stanford colleague John Ioannidis has famously argued that “most published research findings are false”30—and I believe that his argument is largely correct.
What Ioannidis demonstrated was that underpowered studies are not just unlikely to find an effect if it truly exists; any positive findings that they do report are also likely to be false
Since statistical power is relative to the size of the effect we are searching for, there is no single sample size that we can say counts as “big enough,” and this also differs across types of experiments. Unfortunately, there are still many studies published in peer-reviewed journals with sample sizes that are far too small, which means that you need to read closely in order to determine whether any particular result is believable or not. Ultimately, we also want to see that the result can be replicated by other research groups.
By “willpower,” most people think of a specific aspect of self-control that involves either saying no to something that they want (like an extra serving of dessert) or saying yes to something that they don’t want (like going to the gym). It has long been assumed that people with “good willpower” are those who are good at saying no to their impulses in the heat of the moment—overriding their craving for a cigarette or actively choosing the carrot rather than the slice of cake. However, there is increasing evidence that this view is just plain wrong
When we computed the correlation between the stop-signal reaction time (which quantifies how long it takes a person to stop) and a measure of self-control based on questions like the ones listed at the beginning of the chapter, we found virtually no relationship. In fact, across many different measures we found almost zero relationship between tasks meant to measure executive control and surveys meant to measure self-control, just as a number of studies from other researchers have also found
It appears that, instead of being better at inhibiting their impulses, people who appear to have better self-control are actually better at avoiding the need to exert self-control to begin with
Once all of the data were in, Hofmann looked at how an individual’s reported level of self-control related to each of the different aspects that were recorded by experience sampling. If the role of self-control was to squash desires in service of goals, then the people with better self-control should experience more conflict between their desires and their goals and should resist their urges more often. However, the results showed exactly the opposite: the people with higher self-control exhibited less conflict and reported resisting their desires less often than the people with low self-control. In addition, they found that people with higher self-control actually reported experiencing fewer and weaker desires in general.
Another study by Brian Galla and Angela Duckworth from the University of Pennsylvania provides one possible answer to why it is that people who report having higher self-control paradoxically seem to need it less: they are better at establishing good habits. To examine this paradox, they first surveyed a large number of people about their daily habits (including snacking and exercise) and also surveyed their level of self-control. Not surprisingly, they found that people who had better self-control exercised more and ate healthier snacks; but, interestingly, they also reported that their exercise and healthy eating was more habitual, meaning that they just did it automatically without the need to think about it. The researchers also found that the effects of self-control were carried by good habits—better self-control predicted stronger good habits, which in turn predicted less need to exert effortful self-control.42 Similar results were also found in several studies of academic performance and study habits; but perhaps the most interesting finding came from a study they performed that followed 132 individuals going through a five-day meditation retreat. Before the retreat started, Galla and Duckworth measured each individual’s self-control, and then followed up three months later to see how likely the participants were to make a habit of meditation. The people who had higher self-control were more likely to develop a meditation habit after the retreat, and they felt that meditation had become more automatic for them. What this research shows is that willpower is not all that it’s cracked up to be. Next we turn to the question of why some particular habits are so hard to change, where we will also see that willpower doesn’t seem to play the role that many people intuitively believe
neuroscientists reserve the term addiction more specifically for the compulsive and uncontrollable engagement in a particular behavior in spite of its harmful consequences to the user
The process of drug addiction starts with a substance that causes an intoxicating experience, but that experience can vary from the instant peace of heroin to the euphoria of cocaine to the giddiness of alcohol. Despite this variability in the experience, all drugs that are abused by humans appear to cause increases in the level of dopamine, particularly in the nucleus accumbens
Cocaine blocks the activity of the dopamine transporter, while amphetamines can actually cause it to go in reverse, pumping even more dopamine back into the synapse
Other drugs have their effects by causing dopamine neurons to fire more strongly, by either directly causing them to fire (as nicotine and alcohol do) or indirectly causing them to fire by reducing the activity of other cells that normally inhibit the dopamine neurons (as happens with opioids and cannabis)
As we saw in Chapter 2, dopamine neurons will fire for a short time in response to an unexpected reward or to a cue that predicts the later appearance of a reward
it seems that the ability of drugs to drive habit formation may in part reflect the unnatural duration of the dopamine response rather than the amount of dopamine released when the reward is received.
Another study, led by Ilana Witten (whose later work you learned about in Chapter 2), built on decades of research showing that when rats are given the chance to electrically stimulate in their own brains in or near dopamine neurons, they will do so compulsively, in some cases pressing a lever more than 7000 times in an hour
These results show that dopamine stimulation is sufficient to create the kind of compulsive behavior that we often associate with drug abuse
A popular recent idea in the neuroscience of addiction, which has been proposed by the British neuroscientists Trevor Robbins and Barry Everitt,8 is that the development of an addiction involves a transition from impulsivity to compulsivity. This idea proposes that the early experimentation with drugs is related to sensation-seeking tendencies and impulsivity, while the development of compulsive drug use in addiction is related to a transition from goal-directed to habitual behavior.