Trying to figure out how the brain works is like solving a giant jigsaw puzzle. You can approach it in one of two ways. Using the “top-down” approach, you start with the image of what the solved puzzle should look like, and use this to decide which pieces to ignore and which pieces to search for. The other approach is “bottom-up,” where you focus on the individual pieces themselves. You study them for unusual features and look for close matches with other puzzle pieces. If you don’t have a picture of the puzzle’s solution, the bottom-up method is sometimes the only way to proceed.

The “understand-the-brain” jigsaw puzzle is particularly daunting. Lacking a good framework for understanding intelligence, scientists have been forced to stick with the bottom-up approach. But the task is Herculean, if not impossible, with a puzzle as complex as the brain. To get a sense of the difficulty, imagine a jigsaw puzzle with several thousand pieces. Many of the pieces can be interpreted multiple ways, as if each had an image on both sides but only one of them is the right one. All the pieces are poorly shaped so you can’t be certain if two pieces fit together or not. Many of them will not be used in the ultimate solution, but you don’t know which ones or how many. Every month new pieces arrive in the mail. Some of these new pieces replace older ones, as if the puzzle maker was saying, “I know you’ve been working with these old puzzle pieces for a few years, but they turned out to be wrong. Sorry. Use these new ones instead until future notice.” Unfortunately, you have no idea what the end result will look like; worse, you may have some ideas, but they are wrong.

This puzzle analogy is a pretty good description of the difficulty we face in creating a new theory of the cortex and intelligence. The puzzle pieces are the biological and behavioral data that scientists have collected for well over one hundred years. Each month new papers are published, creating additional puzzle pieces. Sometimes the data from one scientist contradict the data from another. Because the data can be interpreted in different ways, there is disagreement over practically everything. Without a top-down framework, there is no consensus on what to look for, what is most important, or how to interpret the mountains of information that have accrued. Our understanding of the brain has been stuck in the bottom-up approach. What we need is a top-down framework.

The memory-prediction model can play this role. It can show us how to start putting pieces of the puzzle together. To make predictions, your cortex needs a way to memorize and store knowledge about sequences of events. To make predictions of novel events, the cortex must form invariant representations. Your brain needs to create and store a model of the world as it is, independent from how you see it under changing circumstances. Knowing what the cortex must do guides us to understanding its architecture, especially its hierarchical design and six-layered form.

As we explore this new framework, presented here for the first time, I will get into a level of detail that may be challenging for some readers. Many of the concepts you are about to encounter are unfamiliar, even to experts in neuroscience. But with a bit of effort, I believe anyone can learn the fundamentals of this new framework. Chapters 7 and 8 of this book are far less technical and explore the wider implications of the theory.

Our puzzle-solving journey can now turn to looking for biological details that support the memory-prediction hypothesis; this is like being able to set aside a large percentage of the puzzle pieces, knowing that the relatively few remaining pieces are going to reveal the ultimate solution. Once we know what to look for, the task becomes manageable.

At the same time, I want to stress that this new framework is incomplete. There are many things I don’t yet understand. But there are many things I do, based on deductive reasoning, experiments carried out in many different laboratories, and known anatomy. In the last five to ten years, researchers from many subspecialties in neuroscience have been exploring ideas similar to mine, although they use different terminology and have not, as far as I know, tried to put these ideas into an overarching framework. They do talk about top-down and bottom-up processing, how patterns propagate through sensory regions of the brain, and how invariant representations might be important. For example, Gabriel Kreiman and Christof Koch, neuroscientists at Caltech, with the neurosurgeon Itzhak Fried at UCLA, have found cells that fire whenever a person sees a picture of Bill Clinton. One of my goals is to explain how those Bill Clinton cells come into being. Of course, all theories need to make predictions that can be tested in the laboratory. I’ve suggested a number of these predictions in the appendix. Now that we know what to look for, this very complex system won’t look so complex anymore.

In the following sections of this chapter, we will probe deeper and deeper into how the memory-prediction model of the cortex works. We will start with the large-scale structure and large-scale function of the neocortex and work toward understanding the smaller pieces and how they fit into the big picture.

Earlier, I painted a picture of the cortex as a sheet of cells the size of a dinner napkin, as thick as six business cards, where the connections between various regions give the whole thing a hierarchical structure. Now I want to draw another picture of the cortex that highlights its hierarchical connectivity. Imagine we cut up the dinner napkin into separate functional regions—sections of cortex that specialize in certain tasks—and stack those regions on top of each other like pancakes. If you cut through this stack and view it from the side, you get figure 1. The cortex doesn’t actually look like this, mind you, but the image will help you visualize how the information is flowing. I have shown four cortical regions in which sensory input enters at the bottom, the lowest region, and flows upward from region to region. Notice information flows both ways.

Figure 1 represents the first four visual regions involved in the recognition of objects—how you come to see and recognize a cat, a cathedral, your mother, the Great Wall of China, you name it. Biologists label them V1, V2, V4, and IT. Visual input represented by the up arrow at the bottom of figure 1 originates in the retinas in both your eyes and is relayed to V1. This input can be thought of as the ever-changing patterns carried on approximately one million axons, bundled together to form your optic nerve.

We talked earlier about spatial and temporal patterns, but it is worth refreshing your memory because we will be referring to them here often. Recall that your cortex is a big sheet of tissue that contains functional areas that specialize in certain tasks. These regions are connected together by large bundles of axons or fibers that transfer information from one region to another, all at once. At any point in time, some set of fibers will fire electrical pulses, again called action potentials or spikes, while others will remain quiet. The collective activity on a bundle of fibers is what is meant by pattern. The pattern arriving at V1 can be spatial, as when your eyes fall for an instant on an object, and temporal, as when your eyes move across the object.

As noted earlier, about three times a second your eyes make a quick movement, called a saccade, and a stop, called a fixation. If a scientist were to fit you with a device that tracks eye movements, you’d be surprised to discover how jerky your saccades are, given that you experience vision as continuous and stable. Figure 2a shows how one person’s eyes moved while looking at a face. Notice how the fixations are not random. Now imagine you could see the pattern of activity arriving at V1 from this person’s eyes. It changes completely with each saccade. Several times a second the visual cortex sees a completely new pattern.

You might think, “OK, but it’s still the same face, just shifted.” There is some truth to this, but not as much as you think. The light receptors in your retina are unevenly distributed. They are

densely concentrated in the fovea at the center, and get gradually sparser out in the periphery. In contrast, the cells in the cortex are evenly distributed. The result is that the retinal image relayed onto the primary visual area, V1, is highly distorted. When your eyes fixate on the nose of a face versus on an eye of the same face, the visual input is very different, as though it is being viewed through a distorting fisheye lens that is jerking violently to and fro. Yet when you see the face, it doesn’t appear distorted, and it doesn’t appear to be jumping around. Most of the time you aren’t even aware that the retinal pattern has changed at all, let alone so dramatically. You just see “face.” (Figure 2b shows this effect on a view of a beach landscape.) This is a restatement of the mystery of invariant representation we talked about in chapter 4, on memory. What you “perceive” is not what V1 sees. How does your brain ever know it is looking at the same face, and why don’t you know the inputs are changing and distorted?

If we stick a probe into V1 and watch how individual cells respond, we find that any particular cell fires only in response to visual input from a tiny part of your retina. This experiment has been done many times and is a mainstay of vision research. Each V1 neuron has a so-called receptive field that is highly specific to a minute part of your total field of vision—that is, the whole world out there in front of your eyes. V1 cells seem to have no knowledge at all about the faces, cars, books, or other meaningful objects you see all the time; all they “know” about is a tiny, pinhole-size portion of the visual world.

Each V1 cell is also tuned for specific kinds of input patterns. For instance, a particular cell might fire vigorously when it sees a line or an edge slanted at thirty degrees within its receptive field. That edge has little meaning in and of itself. It could be part of any object—a floorboard, the trunk of a distant palm tree, the side of a letter M, or any of seemingly infinite other possibilities. With each new fixation, the cell’s receptive field comes to rest on a new and entirely different portion of visual space. On some fixations the cell will fire strongly, on others it will fire weakly or not at all. Thus each time you make a saccade, many cells in V1 are likely to change their activity.

However, something magical happens if you stick a probe into the top region shown in figure 1, region IT. Here we find some cells that become active and stay active when entire objects appear anywhere in the visual field. For example, we might find a cell that fires robustly whenever a face is visible. This cell stays active as long as your eyes are looking at a face anywhere in your field of vision. It doesn’t switch on and off with each saccade as do the cells in V1. This IT cell’s receptive field covers most of the visual space and is tuned to fire when it sees faces.

Let’s recast the mystery. In the course of spanning four cortical stages from retina to IT, cells have changed from being rapidly changing, spatially specific, tiny-feature recognition cells, to being constantly firing, spatially nonspecific, object recognition cells. The IT cell tells us we are seeing a face somewhere in our field of view. This cell, commonly called a face cell, will fire no matter whether the face is tilted, rotated, or partially occluded. It is part of an invariant representation for “face.”

Writing these words makes it seem so simple. Four quick stages and, voilà, we recognize a face. No computer program or mathematical formula has solved this problem with anywhere near the robustness and generality of a human brain. Yet we know the brain solves it in a few steps, so the answer can’t be that difficult. One of the primary goals of this chapter is to explain how a face cell, Bill Clinton’s or otherwise, comes about. We will get there, but we have to cover a lot of other ground first.

Take another look at figure 1. You can see that information also flows from higher to lower regions via a network of feedback connections. These are bundles of axons that go from higher regions like IT to lower regions like V4, V2, and V1. Moreover, there are as many if not more feedback connections in visual cortex as there are feedforward connections.

For many years most scientists ignored these feedback connections. If your understanding of the brain focused on how the cortex took input, processed it, and then acted on it, you didn’t need feedback. All you needed were feedforward connections leading from sensory to motor sections of the cortex. But when you begin to realize that the cortex’s core function is to make predictions, then you have to put feedback into the model; the brain has to send information flowing back toward the region that first receives the inputs. Prediction requires a comparison between what is happening and what you expect to happen. What is actually happening flows up, and what you expect to happen flows down.

The same feedforward-feedback process is occurring in all your cortical areas involving all your senses. Figure 3 shows our

visual pancake stack alongside similar stacks for hearing and touch. It also shows a few higher cortical regions, the association areas, that receive and integrate inputs from a number of different senses—such as hearing plus touch plus vision. While figure 1 is based on known connectivity between four known regions of cortex, figure 3 is a purely conceptual diagram, with no attempt to capture actual cortical regions. In a real human brain dozens of cortical regions are interconnected in all sorts of ways. In fact, the majority of the human cortex consists of association areas. The cartoon characterization shown here and in subsequent figures is meant to help you understand what is going on without misleading you in any significant way.

The transformation—from fast changing to slow changing and from spatially specific to spatially invariant—is well documented for vision. And although there is a smaller body of evidence to prove it, many neuroscientists believe you’d find the same thing happening in all the sensory areas of your cortex, not just in vision.

Take hearing. When someone speaks to you, the changes in sound pressure occur very rapidly; the patterns entering the primary auditory area, called A1, change just as rapidly. Yet if we could poke a probe higher up into your auditory stream, we would find invariant cells that respond to words or even, in some cases, phrases. Your auditory cortex might have a group of cells that fire when you hear “thank you,” and another group of cells that fire for the phrase “good morning.” Such cells would stay active for the whole length of an utterance, assuming that you recognize the phrase.

Patterns received by the first auditory area can vary widely. A word can be spoken with different accents, in different pitches, or at different speeds. But higher up in the cortex, those low-level features don’t matter; a word is a word regardless of the acoustic details. The same thing goes for music. You can hear “Three Blind Mice” played on the piano, on the clarinet, or sung by a child, and your A1 region receives a completely different pattern in each case. But a probe stuck into a higher auditory region should find cells that fire steadily every time “Three Blind Mice” is being played, without concern for the instrument, tempo, or other details. This particular experiment has not been done, of course, since it’s too invasive to perform on humans, but, if you accept that there must be a common cortical algorithm, you can be sure such cells exist. We see the same kind of feedback, prediction, and invariant recall in auditory cortex as we saw in the visual system.

Finally, touch should behave the same way. Again, the definitive experiments have not been done, although research is under way using monkeys in high-resolution brain-imaging machines. As I sit here writing, I have a pen in my hand. I touch the pen’s cap and my finger caresses its metal pocket clip. The patterns entering my somatosensory cortex from the touch sensors in my skin are changing rapidly as my fingers move, yet I perceive a constant pen. At one moment I might flex the metal clip with my fingers, the next moment I do so with a different set of fingers, or even with my lips. These are very different inputs, arriving at different locations in the primary somatosensory cortex. However, our probe would once again find cells in regions several steps removed from the primary input that respond invariantly to “pen.” They would stay active while I fondled the pen and wouldn’t care exactly which fingers or parts of my body I used to touch it.

Think of this. With hearing and touch you can’t recognize an object with a momentary input. The pattern coming from your ears or the touch sensors in your skin does not contain sufficient information at any one point in time to tell you what you are hearing or feeling. When you perceive a series of auditory patterns such as a melody, a spoken word, or a slamming door, and when you perceive a tactile object such as a pen, the only way to do so is by using the flow of input over time. You can’t recognize a melody by hearing one note, and you can’t recognize the feel of a pen with one touch. Therefore the neural activity corresponding to the mental perception of objects, such as spoken words, must last longer in time than the individual input patterns. This is just another way of reaching the same conclusion that the higher up in the cortex you go, the fewer changes over time you should see.

Vision is also a time-based input stream and works in the same general way as hearing and touch, but because we have the ability to recognize individual objects with a single fixation, it confuses the picture. Indeed, this ability to recognize spatial patterns during a brief fixation has for many years misled researchers who work on machine and animal vision. They have generally ignored the critical nature of time. While humans can, under laboratory conditions, be made to recognize objects without moving their eyes, this is not the norm. Normal vision, such as you reading this book, requires constant eye movement.

What about the association areas? So far, we have seen how information flows up and down a particular sensory area of cortex. The downward flow fills in the current input and makes predictions about what we will experience next. The same process occurs between senses—that is, between sight, sound, and touch. For example, something I hear can lead to a prediction of what I should see or feel. Right now I am writing in my bedroom. Our family cat, Keo, has a collar that jingles as she walks. I hear her jingle approaching from the hallway. From this auditory input I recognize my cat, turn my head toward the hallway, and in walks Keo. I expected to see her based on her sound. If Keo hadn’t walked in, or another animal had appeared, I would have been surprised. In this example, an auditory input first created an auditory recognition of Keo. The information flowed up the auditory hierarchy to an association area that connects vision with hearing. The representation then flowed back down the auditory and visual hierarchies, leading to both auditory and visual predictions. Figure 4 illustrates this.

This kind of multisensory prediction is occurring all the time. I bend out the clip on my pen, I feel the clip slip from my fingers, and I expect to hear a snapping sound as the clip hits the barrel of the pen. If I didn’t hear the snap following the release of the clip, I would be surprised. My brain predicts precisely when I will hear the sound and what it will be like. For this prediction to happen, information flowed up the somatosensory cortex and flowed back down both the somatosensory and auditory cortex leading to a prediction of hearing and feeling a snap.

Another example: I ride my bicycle to work several days a week. On those mornings I go into my garage, pick up my bicycle, turn it around, and roll it out into the driveway. In the process of doing so, I receive many visual, tactile, and auditory

inputs. The bicycle hits the doorjamb, the chain rattles, a pedal hits my leg, and the wheel spins as it rubs the floor. In the process of carrying my bicycle out of the garage, my brain encounters a barrage of sight, sound, and touch sensations. Each sensory input stream makes predictions for the others in an amazingly coordinated way. Things I see lead to precise predictions about things I will feel and hear, and the other way around. Seeing the bicycle hit the doorjamb makes me expect to hear a particular sound and feel the bicycle bounce upward. Feeling the pedal hit my leg prompts me to look down and predict to see the pedal right where I felt it. The predictions are so precise that I would notice if any of these inputs were even slightly uncoordinated or unusual. The information simultaneously flows up and down the sensory hierarchies to create a unified sensory experience involving prediction in all senses.

Try this experiment. Get up from reading and do something, any activity that involves moving your body and manipulating an object. For example, go to a sink and turn on the faucet. Now, as you do this, try to notice every sound, touch sensation, and changing visual input. You will have to concentrate. Each action is intimately tied to sights, sounds, and touch sensations. Lift or turn the faucet lever and your brain expects to feel pressure on your skin and resistance in your muscles. You expect to see and feel the lever move, and you expect to see and hear the water. As the water hits the sink, you expect to hear a different sound and to see and feel the splash.

Every footstep makes a sound, which you always anticipate, whether consciously or not. Even the simple act of holding this book leads to many sensory predictions. Imagine if you felt and heard the book close but visually it stayed open. You would be shocked and confused. As we saw with the altered door thought experiment in chapter 5, you make constant predictions about the world that are coordinated across all your senses. When I concentrate on all the little sensations, I am amazed at how fully integrated our perceptual predictions are. Although these predictions may seem simple or trivial, bear in mind how pervasive they are and how they can only come about by massive coordination of patterns streaming up and down the cortical hierarchy.

Once you understand how interconnected the senses are, you are drawn to conclude that the entire neocortex, all the sensory and association areas, acts as one. Yes, we have a visual cortex, but it is just one component of a single, overarching sensory system—sights, sounds, touch, and more combined, all flowing up and down a single multibranched hierarchy.

A further point: All predictions are learned by experience. We expect pen clips to make snapping sounds in the present and future because they have done so in the past. Bicycles banging in garages look, feel, and sound to us in predictable ways. You were not born with any of this knowledge; you learned it thanks to the incredibly large capacity of your cortex for remembering patterns. If there are consistent patterns among the inputs flowing into your brain, your cortex will use them to predict future events.

Although figures 3 and 4 do not depict the motor cortex, you can imagine it as another hierarchical stack of pancakes, just like a sensory stack, connected to the sensory systems via association areas (although with more intimate connections to the somatosensory cortex for carrying out body movements). In this fashion, the motor cortex behaves in almost the same way as a sensory region. An input in any sensory area can flow up to an association area, which can lead to a pattern flowing down the motor cortex, resulting in behavior. Just as a visual input can lead to patterns flowing down the auditory and touch parts of the cortex, it can also lead to a pattern flowing down the motor section of cortex. In the former case, we interpret these downward-flowing patterns as predictions. In the motor cortex we interpret them as motor commands. As Mountcastle pointed out, the motor cortex looks like the sensory cortex. Therefore, the way the cortex processes downward-flowing sensory predictions is similar to how it processes downward-flowing motor commands.

We will see shortly that there are no pure sensory or pure motor areas in the cortex. Sensory patterns simultaneously flow in anywhere and everywhere—and then flow back down any area of the hierarchy, leading to predictions or motor behavior. Although the motor cortex has some special attributes, it is correct to think of it as just part of one large hierarchical memoryprediction system. It’s almost like another sense. Seeing, hearing, touching, and acting are profoundly intertwined.

The next step in untangling the architecture of the cortex requires looking at the cortical regions in a new way. We know the higher regions of cortical hierarchy form invariant representations. But why should this important function only occur at the top? With Mountcastle’s notion of symmetry in the back of my mind, I started to explore the different ways cortical regions might be connected.

Figure 1 portrays the four classic regions of the visual pathway, V1, V2, V4, and IT, with V1 at the bottom of the stack overlain by V2, V4, and, at the top, IT. Conventionally, each is considered and shown as a single, continuous region. Thus all V1 cells supposedly do similar things, although with different parts of the visual field. All of V2’s cells are doing the same kind of task. All V4 cells are similarly specialized.

In this traditional view, when the image of a face enters the V1 region, the cells therein create a rough sketch of the face in terms of simple line segments and other elementary features. The sketch is handed up to V2. Then V2 does its own thing with the image, making a slightly more sophisticated analysis of the facial features, and passes that up to V4, and so on. Invariance, and the recognition of the object, is only achieved when the input reaches the top, IT.

Unfortunately, there are some problems with this view of early cortical regions like V1, V2, and V4. Again, why should invariant representations only show up in IT? If all cortical regions perform the same function, why should IT be special?

Second, a face can appear on the left side of your V1 or on the right side of your V1 and you would recognize it. But experiments clearly show that nonadjacent patches of V1 are not directly connected; the left side of V1 can’t know directly what the right side is seeing. Step back and think about this. Different parts of V1 are obviously doing something similar since they all can participate in recognizing a face, but at the same time they are physically independent. Subregions or clusters of V1 are physically disconnected but do the same thing.

Finally, experiments show that all higher regions of cortex receive converging inputs from two or more sensory regions below themselves (figure 3). In real brains, a dozen regions can converge on an association area. But in traditional renderings, lower sensory regions like V1, V2, and V4 appear to have a different kind of connectivity. Each looks as though it has but one source of input—only one arrow flowing up from the bottom—with no obvious convergence of inputs from different regions. V2 got its input from V1 and that was it. Why should some cortical regions receive converging input and others not? This, too, is inconsistent with Mountcastle’s idea of a common cortical algorithm.

For these and other reasons I have come to believe that V1, V2, and V4 should not be viewed as single cortical regions. Rather, each is a collection of many smaller subregions. Let’s go back to the dinner napkin analogy—a flattened version of the entire cortex. Let’s say we were to use a pen to mark all the functional regions of the cortex on our cortical napkin. The largest region by far is V1, the primary visual area. Next would be V2. They are huge compared to most regions. What I am suggesting is that V1 should actually be considered as many very small regions. Instead of one big area of the napkin, we would draw many small areas that together would occupy the area normally assigned to V1. In other words, V1 is made up of numerous separate little cortical areas that are only connected to their neighbors indirectly, through regions higher up in the hierarchy. V1 would have the largest number of small subregions of any visual area. V2 would also be composed of fewer, slightly larger subregions. The same would be true of V4. But by the time you get to the top region, IT, you really would have a single region, which is why IT cells have a bird’s-eye view of the entire visual world.

There is a pleasing symmetry here. Take a look at figure 5,

which shows the same hierarchy as figure 3, except it shows the sensory hierarchies as I just described. Note that now the cortex looks similar everywhere. Pick any region and you will find many lower regions providing converging sensory input. The receiving region sends projections back to its input regions telling them what patterns they should expect to see next. Higher association areas unite information from multiple senses such as vision and touch. A lower region like a subregion of V2 unites the information from separate subregions within V1. A region doesn’t know—indeed it can’t know—what any of those inputs mean. A V2 subregion doesn’t need to know that it is handling visual input from multiple parts of V1. An association area doesn’t need to know that it is handling input from vision and hearing. Rather, the job of any cortical region is to find out how its inputs are related, to memorize the sequence of correlations between them, and to use this memory to predict how the inputs will behave in the future. Cortex is cortex. The same process is happening everywhere: a common cortical algorithm.

This new hierarchical depiction helps us understand the process of creating invariant representations. Let’s look more closely at how it works in vision. At the first level of processing, the left side of visual space is different from the right side of visual space in the same way that hearing is different from seeing. Left V1 and right V1 form the same kind of representations only because they have been exposed to similar patterns in life. Like hearing and seeing, they can be viewed as separate sensory streams, which get united higher up.

Similarly, the small regions within V2 and V4 are association areas of vision. (Subregions may overlap, but this would not fundamentally change the way these regions work.) Interpreting the visual cortex this way does not contradict or change anything we know about its anatomy. Information flows up and down all branches of the hierarchical memory tree. A pattern in the left visual field can lead to a prediction in the right visual field in exactly the same way that my cat’s bell can lead to a visual prediction that she is entering my bedroom.

The most important result of this new depiction of the cortical hierarchy is that now we can say each and every region of cortex forms invariant representations. In the old way of thinking, we didn’t have complete invariant representations—such as faces—until inputs reached the top layer, IT, which sees the whole visual world. Now we can say that invariant representations are ubiquitous. Invariant representations are formed in every cortical region. Invariance isn’t something that only magically appears when we get to higher regions of the cortex, such as IT. Every region forms invariant representations drawn from the input areas hierarchically below it. Thus the subregions of V4, V2, and V1 create invariant representations based on what flows into them. They may only see a tiny part of the world, and the vocabulary of sensory objects they deal with is more basic, but they are performing the same job as IT. Also, association regions above IT form invariant representations of patterns from multiple senses. Thus all regions of cortex form invariant representations of the world underneath them in the hierarchy. There is beauty in this.

Our puzzle has shifted. We no longer have to ask how invariant representations are formed in four steps from bottom to top. Rather we have to ask how invariant representations are formed in every single cortical region. This makes perfect sense if we take the existence of a common cortical algorithm seriously. If one region stores sequences of patterns, every region stores sequences. If one region creates invariant representations, all regions create invariant representations. Redrawing the cortical hierarchy along the lines shown in figure 5 makes this interpretation possible.

Why is the neocortex built as a hierarchy?

You can think about the world, move around in the world, and make predictions of the future because your cortex has built a model of the world. One of the most important concepts in this book is that the cortex’s hierarchical structure stores a model of the hierarchical structure of the real world. The real world’s nested structure is mirrored by the nested structure of your cortex.

What do I mean by a nested or hierarchical structure? Think about music. Notes are combined to form intervals. Intervals are combined to form melodic phrases. Phrases are combined to form melodies or songs. Songs are combined into albums. Think about written language. Letters are combined to form syllables. Syllables are combined to form words. Words are combined to form clauses and sentences. Looking at it the other way around, think about your neighborhood. It probably contains roads, schools, and houses. Houses have rooms. Each room has walls, a ceiling, a floor, a door, and one or more windows. Each of these is composed of smaller objects. Windows are made of glass, frames, latches, and screens. Latches are made from smaller parts like screws.

Take a moment to look up at your surroundings. Patterns from the retina entering your primary visual cortex are being combined to form line segments. Line segments combine to form more complex shapes. These complex shapes are combining to form objects like noses. Noses are combining with eyes and mouths to form faces. And faces are combining with other body parts to form the person who is sitting in the room across from you.

All objects in your world are composed of subobjects that occur consistently together; that is the very definition of an object. When we assign a name to something, we do so because a set of features consistently travels together. A face is a face precisely because two eyes, a nose, and a mouth always appear together. An eye is an eye precisely because a pupil, an iris, an eyelid, and so on, always appear together. The same can be said for chairs, cars, trees, parks, and countries. And, finally, a song is a song because a series of intervals always appear together in sequence.

In this way the world is like a song. Every object in the world is composed of a collection of smaller objects, and most objects are part of larger objects. This is what I mean by nested structure. Once you are aware of it, you can see nested structures everywhere. In an exactly analogous way, your memories of things and the way your brain represents them are stored in the hierarchical structure of the cortex. Your memory of your home does not exist in one region of cortex. It is stored over a hierarchy of cortical regions that reflect the hierarchical structure of the home. Large-scale relationships are stored at the top of the hierarchy and small-scale relationships are stored toward the bottom.

The design of the cortex and the method by which it learns naturally discover the hierarchical relationships in the world.

You are not born with knowledge of language, houses, or music. The cortex has a clever learning algorithm that naturally finds whatever hierarchical structure exists and captures it. When structure is absent, we are thrown into confusion, even chaos.

You can only experience a subset of the world at any moment in time. You can only be in one room of your home, looking in one direction. Because of the hierarchy of the cortex, you are able to know that you are at home, in your living room, looking at a window, even though at that moment your eyes happen to be fixated on a window latch. Higher regions of cortex are maintaining a representation of your home, while lower regions are representing rooms, and still lower regions are looking at a window. Similarly, the hierarchy allows you to know you are listening to both a song and an album of music, even though at any point in time you are hearing only one note, which on its own tells you next to nothing. It allows you to know you are with your best friend, even though your eyes are momentarily fixated on her hand. Higher regions of your cortex are keeping track of the big picture while lower areas are actively dealing with the fast-changing, small details.

Since we can only touch, hear, and see a very small part of the world at any moment in time, information flowing into the brain naturally arrives as a sequence of patterns. The cortex wants to learn those sequences that occur over and over again. In some cases, such as melodies, a sequence of patterns comes in a rigid order, the order of the intervals. Most of us are familiar with that kind of sequence. But I am going to use the word sequence in a more general way, closer in meaning to the mathematical term set. A sequence is a set of patterns that generally accompany each other but not always in a fixed order. What is important is that patterns of a sequence follow one another in time even if not in a fixed order.

Some examples should make this clear. When I look at your face, the sequence of input patterns I see is not fixed but is determined by my saccades. One time I might fixate in the order “eye eye nose mouth,” and a moment later fixate in the order “mouth eye nose eye.” The components of a face are a sequence. They are statistically related and tend to occur together in time, although the order may vary. If you perceive “face” while fixating on “nose,” the likely next patterns would be “eye” or “mouth” but not “pen” or “car.”

Each region of cortex sees a stream of such patterns. If the patterns are related in such a way that the region can learn to predict what pattern will occur next, the cortical region forms a persistent representation, or memory, for the sequence. Learning sequences is the most basic ingredient for forming invariant representations of real-world objects.

Real-world objects can be concrete, like a lizard, a face, or a door, or they can be abstract, like a word or a theory. The brain treats abstract and concrete objects in the same way. They are both just sequences of patterns that occur together over time in a predictable fashion. The fact that certain input patterns repeat time and again is what lets a cortical region know that those experiences are caused by a real object in the world.

Predictability is the very definition of reality. If a region of cortex finds it can reliably and predictably move among these input patterns using a series of physical motions (such as saccades of the eyes or fondling with the fingers) and can predict them accurately as they unfold in time (such as the sounds comprising a song or a spoken word), the brain interprets these as having a causal relationship. The odds of numerous input patterns occurring in the same relation over and over again by sheer coincidence are vanishingly small. A predictable sequence of patterns must be part of a larger object that really exists. So reliable predictability is an ironclad way of knowing that different events in the world are physically tied together. Every face has eyes, ears, mouth, and nose. If the brain sees an eye, then saccades and sees another eye, then saccades and sees a mouth, it can feel certain it is seeing a face.

If cortical regions could speak they might say, “I experience many different patterns. Sometimes I can’t predict what pattern I will see next. But these patterns are definitely related to one another. They always occur together, and I can reliably jump between them. So whenever I see any of these events, I will refer to them by a common name. It is this group name, not the individual patterns, that I will pass on to higher regions of the cortex.”

Therefore, the brain can be said to store sequences of sequences. Each region of the cortex learns sequences, develops what I call “names” for the sequences it knows, and passes these names to the next regions higher in the cortical hierarchy.

As information moves up from primary sensory regions to higher levels, we see fewer and fewer changes over time. In primary visual areas like V1, the set of active cells is changing rapidly as new patterns fall on the retina several times each second. In visual area IT, cell firing patterns are more stable. What is happening here? Each region of cortex has a repertoire of sequences it knows, analogous to a repertoire of songs. Regions store these songlike sequences about anything and everything: the sound of surf crashing on the beach, your mother’s face, the path from your home to the corner store, how to spell the word “popcorn,” how to shuffle a deck of cards.

We have names for songs, and in a similar fashion each cortical region has a name for each sequence it knows. This “name” is a group of cells whose collective firing represents the set of objects in the sequence. (Never mind for now how that group of cells is selected to represent the sequence; we will get to this later.) These cells remain active as long as the sequence is playing, and it is this “name” that is passed up to the next region in the hierarchy. As long as the input patterns are part of a predictable sequence, the region presents a constant “name” to the next higher region.

It’s as if the region were saying, “Here is the name of the sequence that I am hearing, seeing, or touching. You don’t need to know about the individual notes, edges, or texture. I will let you know if something new or unpredicted happens.” More specifically, we can imagine region IT at the top of the visual hierarchy relaying to an association area above it, “I am seeing a face. Yes, with each saccade the eyes are fixating on different parts of the face; I am seeing different parts of the face in succession. But it is still the same face. I will let you know when I see something else.” In this fashion, a predictable sequence of events gets identified with a “name”—a constant pattern of cell firing. This happens over and over again as we go up the hierarchical pyramid. One region might recognize a sequence of sounds that comprise phonemes (the sounds that make up words) and passes a pattern representing the phoneme up to the next region. The next higher region recognizes sequences of phonemes to create words. The next higher region recognizes sequences of words to create phrases, and so on. Bear in mind that a “sequence” in the lowest regions of cortex may be fairly simple, such as a visual edge moving through space.

By collapsing predictable sequences into “named objects” at each region in our hierarchy, we achieve more and more stability the higher we go. This creates invariant representations.

The opposite effect happens as a pattern moves back down the hierarchy: stable patterns get “unfolded” into sequences. Let’s assume you memorized the Gettysburg Address when you were in the seventh grade and now you want to recite it. In a language region high up in your cortex, there is a stored pattern that represents Lincoln’s famous speech. First, this pattern is unfolded into a memory of the sequence of phrases. In the next region down, each phrase is unfolded into a memory of the sequence of words. At this point, the unfolding pattern splits and travels down both the auditory section of cortex and the motor section of cortex. Following the motor path, each word is unfolded into a memorized sequence of phonemes. And in the final, bottom region, each phoneme is unfolded into a sequence of muscle commands to make sounds. The lower you look in the hierarchy, the faster the patterns are changing. A single, constant pattern at the top of your motor hierarchy eventually leads to a complex and lengthy sequence of speech sounds.

Invariance also works to our advantage as this information flows back down the hierarchy. If you want to type the Gettysburg Address instead of speak it, you start out with the same pattern at the top of your hierarchy. The pattern is unfolded into phrases in the next region down. The phrases are unfolded into words in the region below that. So far there is no difference between speaking and typing the Gettysburg Address. But at the next level down, your motor cortex takes a different path. Words are unfolded into letters, and the letters are unfolded into muscle commands to your fingers for typing. “Four score and seven years ago our fathers brought forth …” The memories of the words are handled as invariant representations; it doesn’t matter whether you will speak, type, or handwrite them. Notice you don’t have to memorize the speech twice, once for speaking and once for writing. A single memory of the speech can take various behavioral forms. In any region, an invariant pattern can bifurcate and follow a different path down.

In a complementary bit of efficiency, representations of simple objects at the bottom of the hierarchy can be reused over and over for different high-level sequences. For instance, we don’t have to learn one set of words for the Gettysburg Address and a completely different set for Martin Luther King’s “I Have a Dream” speech, even though the two orations contain some of the same words. A hierarchy of nested sequences allows the sharing and reuse of lower-level objects—words, phonemes, and letters being but a few examples. It is a remarkably efficient way to store information about the world and its structure and very different from how computers work.

The same unfolding of sequences occurs in the sensory as well as the motor regions. The process allows you to perceive and understand objects from different views. If you are walking up to your refrigerator to get some ice cream, your visual cortex is active on multiple levels. At a higher level, you are perceiving a constant “refrigerator.” In lower regions, this visual expectation is broken down into a series of more localized visual inputs. Seeing the refrigerator is composed of fixations on the door handle, the ice dispenser, the magnets on the door, a child’s drawing, and so forth. In the few milliseconds that elapse as you saccade from one feature of your refrigerator to another, predictions about the result of each saccade are cascading down your visual hierarchy. As long as these predictions get confirmed saccade after saccade, your higher visual regions remain satisfied that you are in fact looking at your refrigerator. Note that in this case, unlike the fixed order of words in the Gettysburg Address, the sequence you see when looking at the refrigerator is not fixed; the flow of inputs and retrieved memory patterns depend on your own actions. So in a case like this, the unfolding pattern is not a rigid sequence, but the end result is the same: slowchanging, high-level patterns unfolding into faster-changing, low-level patterns.

The way you memorize sequences and represent them by name as information goes up and down your cortical hierarchy may remind you of the hierarchy of military command. The top army general says, “Move the troops to Florida for the winter.” This simple high-level command gets unfolded into ever more detailed sequences of commands as it percolates down the hierarchy. The general’s underlings recognize that the command requires a sequence of steps such as preparations to leave, transportation to Florida, and preparations for arrival. Each of these steps breaks down into further, more specific steps, to be carried out by subordinates. At the bottom there are thousands of privates taking tens of thousands of actions that result in the troops moving. Reports of what happened are generated at each level. As they percolate back up the hierarchy, they are summarized again and again, until at the very top of the hierarchy, the general receives a daily briefing saying, “Move to Florida going okay.” The general does not get all the details.

There is an exception to this rule. If something goes wrong that cannot be handled by subordinates down the chain of command, then the issue rises up the hierarchy until someone knows what to do next. The officer who does know how to handle the situation does not see it as an exception. What was an unanticipated problem to subordinates is just the expected next task on his list. The officer then issues new commands to subordinates. The neocortex behaves similarly. As we will see in a bit, when events (in other words, patterns) occur that aren’t anticipated, information about them progresses up the cortical hierarchy until some region can handle it. If lower regions of cortex fail to predict what patterns they are seeing, they consider this an error and pass the error up the hierarchy. This is repeated until some region does anticipate the pattern.

By design, every cortical region attempts to store and recall sequences. But this is still too simple a description of the brain. We need to add a few more complexities into the model.

The bottom-up inputs to a region of cortex are input patterns carried on thousands or millions of axons. These axons come from different regions and contain all sorts of patterns. The number of possible patterns that can exist on even one thousand axons is larger than the number of molecules in the universe. A region will only see a tiny fraction of these possible patterns in a lifetime.

So here’s a question: When a single region stores sequences, what are they sequences of? The answer is that a region first classifies its inputs as one of a limited number of possibilities, and then looks for sequences. Imagine that you are a single cortical region. Your task is to sort colored pieces of paper. You are supplied ten buckets, each of which is labeled with a sample color swatch. There is one bucket for green, one for yellow, another for red, and so on. You are then given pieces of colored paper, one by one, and told to sort them by color. Each paper you receive is slightly different. Because there are an infinite number of colors in the world, you never get two pieces of paper with exactly the same color. Sometimes it is easy to say what bucket the colored paper should be placed in, but sometimes it is difficult. A paper that is halfway between red and orange could go in either bucket, but you have to assign it to one bucket, either red or orange, even if the selection comes down to a random pick. (The point of this exercise is to show that the brain must classify patterns. Regions of cortex do this, but there is nothing equivalent to a bucket to put patterns in.)

Now you are given the additional task of looking for sequences. You notice that red-red-green-purple-orange-green occurs frequently. You call it the “rrgpog” sequence. Notice that recognizing any sequence would be impossible if you hadn’t first classified each piece of paper. Without first classifying each piece of paper into one of ten categories, you couldn’t say that two sequences are the same.

So now you are up and running. You are going to look at all the input patterns—the colored pieces of paper coming in from lower cortical regions—classify them, and then look for sequences. Both steps, classification and sequence formation, are necessary to create invariant representations, and each region of cortex does them.

The process of forming sequences pays off when an input is ambiguous, like a piece of paper that falls somewhere between red and orange. You have to pick a bucket for the paper even if you’re not sure if it’s more red or more orange. If you know the most likely sequence for this series of inputs, you will use this knowledge to decide how to classify the ambiguous input. If you believe you are in the “rrgpog” sequence because you have just gotten two reds, a green, and a purple, you’re going to expect that the next paper will be orange. But the next piece of paper arrives and it’s not orange. Rather, it is an odd color somewhere between red and orange. It might even be a little more red than orange. But you are familiar with and expecting the “rrgpog” sequence, and so you place the paper in the orange bucket. You use the context of known sequences to resolve ambiguity.

We see this phenomenon happening all the time in our everyday experiences. When people speak, their individual words very often cannot be understood out of context. Yet when you hear an ambiguous word in a sentence, you don’t get hung up on the word’s ambiguity. You understand it. Similarly, handwritten words are often unintelligible out of context, but very readable within a full written sentence. Most of the time you aren’t aware that you are filling in ambiguous or incomplete information from your memories of sequences. You hear what you expect to hear and see what you expect to see—at least when what you hear and see fits into past experience.

Notice the memory of sequences allows you not only to resolve ambiguity in the current input, but also to predict which input should happen next. While your cortical self is sorting colored papers, you can tell the “input” person passing the papers to you, “Hey, in case you are having any trouble deciding what to pass me next, it should, according to my memory, be an orange one.” By recognizing a sequence of patterns, a cortical region will predict its next input pattern and tell the region below what to expect.

A region of cortex not only learns familiar sequences, it also learns how to modify its classifications. Let’s say you start out with a set of buckets labeled “green,” “yellow,” “red,” “purple,” and “orange.” You are prepared to recognize the “rrgpog” sequence as well as other combinations of these colors. But what if a color makes a major shift? What if every time you see the “rrgpog” sequence, the purple is way off-kilter? The new color is more like indigo. So you change the purple bucket to be the “indigo” bucket. Now the buckets better fit what you see; you have reduced the ambiguity. The cortex is flexible.

In cortical regions, bottom-up classifications and top-down sequences are constantly interacting, changing throughout your life. This is the essence of learning. In fact, all regions of the cortex are plastic, thus they can be modified by experience. Forming new classifications and new sequences is how you remember the world.

Last, let’s look at how these classifications and predictions interact with the next higher region. Another part of your cortical job is to relay the name of the sequence you are seeing to the next level up, so you pass up a piece of paper with the letters “rrgpog” on it. These letters mean little in themselves to the next higher region; the name is just a pattern to be combined with other inputs, classified, and then put into yet a higher-order sequence. Like you, he is keeping track of the sequences he is seeing. At some point he might say to you, “Hey, in case you are having any trouble deciding what to pass me next, according to my memory, I predict it should be the ‘yyrgy’ sequence.” This is basically an instruction to you about what to look for in your own input stream. You will do your best to interpret what you see as comprising that sequence.

Since many people have heard the term pattern classification used in AI and machine vision research, let’s look at how this process as it is usually understood differs from what the cortex is doing. In trying to get machines to recognize objects, researchers typically create a template—say the image of a cup, or some prototypical form of cup—and then instruct the machine to match its inputs with the prototypical cup. If it finds a close match, the computer will say it has found a cup. But our brains don’t have templates like that, and the patterns that each cortical region receives as input aren’t like pictures. You don’t remember snapshots of what your retina sees, or snapshots of the patterns from your cochlea or your skin. The hierarchy of the cortex ensures that memories of objects are distributed over the hierarchy; they aren’t located in a single spot. Also, because each region of the hierarchy forms invariant memories, what a typical region of cortex learns is sequences of invariant representations, which are themselves sequences of invariant memories. You won’t find a picture of a cup or any other object stored in your brain.

Unlike a camera’s memory, your brain remembers the world as it is, not as it appears. When you think about the world, you are recalling sequences of patterns that correspond to the way the objects in the world are and how they behave, not how they appear through any particular sense at any point in time. The sequences by which you experience objects in the world reflect the invariant structure of the world itself. The order in which you experience parts of the world is determined by the world’s structure. For example, you can get onto an airplane directly by walking down a jetway, but not from the ticket counter. The sequences by which you experience the world is the real structure of the world, and that is what the cortex wants to remember.

Don’t forget, though, an invariant representation in any region of the cortex can be turned into a detailed prediction of how it will appear on your senses by propagating the pattern down the hierarchy. Similarly, an invariant representation in the motor cortex can be turned into detailed and situation-specific motor commands by propagating the pattern down the motor hierarchy.

We are now going to turn our attention to an individual region of cortex, one of the boxes in figure 5. Figure 6 shows such a

region of cortex in more detail. My goal is to show you how the cells in a region of cortex can learn and recall sequences of patterns, which is the most essential element for forming invariant representations and making predictions. We will start with a description of what a cortical region looks like, and how it is put together. Cortical regions vary greatly in size, the largest being the primary sensory areas. V1, for example, is roughly the size of a passport in terms of the space it occupies at the back of the brain. But as I argued earlier, it is actually composed of many smaller regions that might be the size of the letters on this page. For now, let’s assume that a typical cortical area is the size of a small coin.

Think of the six business cards I mentioned in chapter 3, where each card represents a different layer of cortical tissue. Why do we say there are layers? If you take our coin-size region of cortex and place it under a microscope, you’ll see that the density and shape of the cells vary as you move from top to bottom. These differences define the layers. The top, called layer 1, is the most distinct of the six layers. It has very few cells, consisting primarily of a mat of axons running parallel to the cortical surface. Layers 2 and 3 look similar. They contain many tightly packed pyramidal cells. Layer 4 has a type of star-shaped cell. Layer 5 has regular pyramidal cells as well as a class of extra-big pyramid-shaped cells. The bottom layer, layer 6, also has several types of unique neurons.

Visually we see horizontal layers, but most often scientists talk about columns of cells that run perpendicular to the layers. You can think of columns as being vertical “units” of cells that work together. (The term column invites much debate in the neuroscience community. Their size, function, and importance are disputed. For our purposes, though, you can think in general terms of a columnar architecture, which everyone agrees exists.) The layers within each column are connected via axons that run up and down, making synapses along the way. Columns do not stand out like neat little pillars with clear boundaries—nothing in the cortex is that simple—but their existence can be inferred from several lines of evidence.

One reason is that the vertically aligned cells in each column tend to become active for the same stimulus. If we looked closely at columns in V1, we’d find some that respond to line segments that tilt in one direction (/) and some that respond to line segments that tilt in another direction (\). The cells within each column are strongly connected, which is why the entire column responds to the same stimulus. Specifically, an active cell in layer 4 causes cells above it in layers 3 and 2 to become active, which then cause cells below in layers 5 and 6 to become active. Activity spreads up and down within a column of cells.

Another reason we talk of columns stems from how the cortex forms. In an embryo, single precursor cells migrate from an inner brain cavity to where the cortex takes shape. Each of these cells divides to create about one hundred neurons, called a microcolumn, which are connected in the vertical fashion I just described. The term column is often used loosely to describe different phenomena; it can refer to general vertical connectivity or to specific groups of cells from the same progenitor. Using the latter definition, we can say that the human cortex has an estimated several hundred million microcolumns.

To help you visualize this columnar structure, imagine a single microcolumn is the width of a human hair. Take thousands of hairs and cut them into very short segments—say, the height of a lowercase i without the dot. Line up all these hairs or columns and glue them side by side like a very dense brush. Then create a sheet of long, extra-thin hairs—representing your layer 1 axons—and glue them horizontally across the top of the mat of short hairs. This brushlike mat is a simplistic model of your coin-size cortical region. Information flows mostly in the direction of these hairs: horizontally in layer 1 and vertically in layers 2 through 6.

There’s one more detail about columns you need to know, and then we’ll move on to what they’re good for. On close inspection, we see that at least 90 percent of the synapses on cells within each column come from places outside the column itself. Some connections arrive from neighboring columns. Others come from halfway across the brain. How, then, can we speak of columns as being important when so much of our cortical wiring is spread out laterally over large areas?

The answer is in the memory-prediction model. In 1979, when Vernon Mountcastle argued that there is a single cortical algorithm, he also proposed that the cortical column is the basic unit of computation in the cortex. However, he did not know what function a column performs. I believe that a column is the basic unit of prediction. For a column to predict when it should be active, it needs to know what is going on elsewhere—hence the synaptic connections from hither and yon.

We’ll get into more details soon, but here’s a preview of why we need this sort of wiring in the brain. To predict the next note of a song, you need to know the song’s name, where you are in the song, how much time has passed since the last note, and what that last note was. The large number of synapses connecting cells in a column to other parts of the brain provide each column with the context it needs in order to predict its activity in many different situations.

The next thing we need to consider is how these coin-size cortical regions (and their columns) send and receive information up and down the cortical hierarchy. We’ll look at the upward flow first, which takes a relatively direct route, depicted in figure 7. Imagine we’re looking at a cortical region with its thousands of columns. We zoom in on just one. Converging inputs from lower regions always arrive at layer 4—the main input layer. In passing, they also form a connection in layer 6 (we’ll see later why this is important). Layer 4 cells then send projections up to cells in layers 2 and 3 within their column. When a column projects information up, many layer 2 and layer 3 cells send axons to the input layer of that next higher region. Thus information flows from region to region up the hierarchy.

Information flowing down the cortical hierarchy takes a less direct path, as depicted in figure 8. Layer 6 cells are the downward-projecting output cells from a cortical column and project to layer 1 in the regions hierarchically below. Here in

layer 1, the axon spreads over long distances in the lower cortical region. Thus information flowing down the hierarchy from one column has the potential to activate many columns in the regions below it. There are very few cells in layer 1, but cells in layers 2,3, and 5 have dendrites in layer 1, so these cells can be excited by the feedback running all across layer 1. The axons coming from layers 2 and 3 cells form synapses in layer 5 as they leave the cortex and are believed to excite cells in layers 5 and 6. So we can say that as information flows down the hierarchy, it has a less direct route. It can branch in many different directions via the spread on layer 1. Feedback information starts in a layer 6 cell in the higher region; it spreads across layer 1 in the lower region. Some cells in layers 2, 3, and 5 in the lower region are excited, and some of these excite layer 6 cells, which project to layer 1 in regions hierarchically below, and so on. (If you study figure 8, the process is much easier to follow.)

Here is a preview of why information is spread across layer 1.

To convert an invariant representation into a specific prediction requires the ability to decide moment by moment which way to send the signal as it propagates down the hierarchy. Layer 1 provides a way of converting an invariant representation into a more detailed and specific representation. Remember that you can recall the Gettysburg Address either in spoken language or in writing. A common representation is moved along one of two paths, one for speaking and one for writing. Similarly when I hear the next note in a melody, my brain has to take a generic interval such as a fifth and convert it to the correct specific note, such as C or G. The horizontal flow of activity across layer 1 provides the mechanism for doing this. For high-level invariant predictions to propagate down the cortex and become specific predictions, we must have a mechanism that allows the flow of patterns to branch at each level. Layer 1 fits the bill. We could predict the need for it even if we didn’t know it existed.

One final bit of anatomy: when axons leave layer 6 to travel to other destinations, they are encased in a white fatty substance called myelin. This so-called white matter is like the insulation on an electrical wire in your house. It helps prevent signals from getting mixed up and makes them travel faster, at speeds up to two hundred miles per hour. When axons leave the white matter, they enter a new cortical column at layer 6.

Finally, there is another, indirect method for cortical regions to communicate with each other.

Before I describe the details, I want to remind you of the auto-associative memories discussed in chapter 2. As you may recall, auto-associative memories can be used to store sequences of patterns. When the output of a group of artificial neurons is fed back to form the input to all the neurons, and a delay is added to the feedback, then patterns learn to follow each other in sequence. I believe the cortex uses the same basic mechanism to store sequences, although with a few extra twists. Instead of forming an auto-associative memory out of artificial neurons, it is forming an auto-associative memory out of cortical columns. The output of all the columns is fed back to layer 1. In this way, layer 1 contains information about which columns were just active in the region of cortex.

Let’s walk through the elements, as shown in figure 9. It has been known for many years that particularly large layer 5 cells within your motor cortex (region Ml) make direct contact with your muscles and the motor regions of your spinal cord. These cells literally drive your muscles and make you move. Whenever

you speak, type, or perform any sophisticated behavior, these cells are firing on and off in a highly coordinated way, making your muscles contract.

Recently, researchers have discovered that large layer 5 cells may play a role in behavior in other parts of the cortex, not just in motor regions. For example, large layer 5 cells in visual cortex project to the part of the brain that moves the eyes. So the sensory visual areas of cortex, such as V2 and V4, not only process visual input, but they help determine the movement of the eyes themselves, and therefore what you see. Large layer 5 cells are seen throughout the neocortex, in every region, suggesting a more widespread role in all kinds of movement.

In addition to having a behavioral role, the axons of these large layer 5 cells split in two. One branch goes to the part of the brain called the thalamus, shown as the round object in figure 9. The human thalamus is the shape and size of two small bird eggs. It sits in the very center of the brain, on top of the old brain, and is surrounded by white matter and cortex. The thalamus receives many axons from every part of the cortex and sends axons back to those same areas. Many of the details of these connections are known, but the thalamus itself is a complicated structure and its role is not at all clear. But the thalamus is essential to normal living; a damaged thalamus leads to a persistent vegetative state.

There are a couple of paths from the thalamus to the cortex, but only one is of interest to us now. This path starts with the large layer 5 cells that project to a class of thalamic cells considered nonspecific. The nonspecific cells project axons back to layer 1 over many different regions of cortex. For example, layer 5 cells throughout the V2 and V4 regions send axons to the thalamus, and the thalamus sends the information back to layer 1 throughout V2 and V4. Other parts of the cortex do the same; layer 5 cells throughout multiple cortical regions project to the thalamus, which sends the information back to layer 1 of those same and associated regions. I propose this circuit is exactly like the delayed feedback that lets auto-associative memory models learn sequences.

I have now mentioned two inputs to layer 1. Higher regions of cortex spread activity across layer 1 in lower regions. Active columns within a region also spread activity across layer 1 in the same region via the thalamus. We can think of these inputs to layer 1 as the name of a song (input from above) and where we are in a song (delayed activity from active columns in the same region). Thus layer 1 carries much of the information we need to predict when a column should be active—the sequence name and where we are within the sequence. Using these two signals in layer 1, a region of cortex can learn and recall multiple sequences of patterns.

With these three circuits in mind—converging patterns going up the cortical hierarchy, diverging patterns going down the cortical hierarchy, and a delayed feedback through the thalamus—we can start to see how a region of cortex performs the functions it needs. What we want to know is:

1. How does a region of cortex classify its inputs (like the buckets)?

2. How does it learn sequences of patterns (such as intervals of a melody, or the “eye nose eye” of a face)?

3. How does it form a constant pattern or “name” for a sequence?

4. How does it make specific predictions (meeting the train at the right time, or predicting the specific note in a melody)?

Let’s start by assuming that the columns in a region of cortex are like the buckets we used in classifying our colored paper inputs. Each column represents the label of a bucket. The layer 4 cells in every column receive input fibers from several regions below it and will fire if they have the right combination of inputs. When a layer 4 cell fires, it is “voting” that the input fits its label. As in the paper-sorting analogy, inputs can be ambiguous, so several columns might be possible matches for the input. We want our region of cortex to decide on one interpretation; the paper is either red or orange, but not both. A column with strong input should prevent other columns from firing.

Brains have inhibitory cells that do just this. They strongly inhibit other neurons in a neighborhood of cortex, effectively allowing one winner. These inhibitory cells only affect the area surrounding a column. So even though there is a lot of inhibition, many columns in a region can still be active at once. (In real brains, nothing is ever represented by a single neuron or a single column.) To make it easier to follow, you can pretend that a region picks one and only one winner column. But in the back of your mind, remember that many columns will be active at once. The actual process used by a region of cortex to classify inputs and how it learns to do this is complicated, and not well understood. I will not attempt to drag you through the issues. Instead I want to assume that our region of cortex has classified its input as activity in a set of columns. We can then focus on the forming of sequences and names for sequences.

How does our cortical region store the sequence of these classified patterns? I have already suggested an answer to this question, but now I’ll delve into more details. Imagine you are a column of cells, and input from a lower region causes one of your layer 4 cells to fire. You are happy, and your layer 4 cell causes cells in layers 2 and 3, then 5, and then 6 also to fire. The entire column becomes active when driven from below. Your cells in layers 2, 3, and 5 each have thousands of synapses in layer 1. If some of these synapses are active when your layer 2,3, and 5 cells fire, the synapses are strengthened. If this occurs often enough, these layer 1 synapses become strong enough to make the cells in layers 2,3, and 5 fire even when a layer 4 cell hasn’t fired—meaning parts of the column can become active without receiving input from a lower region of the cortex. In this way, cells in layers 2, 3, and 5 learn to “anticipate” when they should fire based on the pattern in layer 1. Before learning, the column can only become active if driven by a layer 4 cell. After learning, the column can become partially active via memory. When a column becomes active via layer 1 synapses, it is anticipating being driven from below. This is prediction. If the column could speak, it would say, “When I have been active in the past, this particular set of my layer 1 synapses has been active. So when I see this particular set again, I will fire in anticipation.”

Recall that half of the input to layer 1 comes from layer 5 cells in neighboring columns and regions of the cortex. This information represents what was happening moments before. It represents columns that were active prior to your column becoming active. It represents the previous interval in the melody, or the last thing I saw, or the last thing I felt, or the previous phoneme in the speech I am listening to. If the order in which these patterns occur over time is consistent, then the columns will learn the order. They will fire one after another in proper sequence.

The other half of the input to layer 1 comes from layer 6 cells in hierarchically higher regions. This information is more stationary. It represents the name of the sequence you are currently experiencing. If your columns are music intervals, it is the name of the melody. If your columns are phonemes, then it is the spoken word you are hearing. If your columns are spoken words, then the signal from above is the speech you are reciting. Thus the information in layer 1 represents both the name of a sequence and the last item in the sequence. In this way, a particular column can be shared among many different sequences without getting confused. Columns learn to fire in the right context and in the correct order.

Before moving on, I need to point out that the synapses in layer 1 are not the only synapses that participate in learning when a column should become active. As I mentioned earlier, cells receive input from, and send input to, many surrounding columns. Recall that more than 90 percent of all synapses are from cells outside the column, and most of these synapses are not in layer 1. For example, cells in layers 2, 3, and 5 have thousands of synapses in layer 1, but also thousands of synapses in their own layers. The overall idea is that cells want any information that will help them predict when they will be driven from below.

Usually, activity in nearby columns has a strong correlation, and so we see many direct connections to nearby columns. For example, if a line is moving through your visual field, it will activate successive columns. Often, though, the information needed to predict a column’s activity is more global, which is where the layer 1 synapses play a role. If you were a cell or a column, you wouldn’t know what any of these synapses meant, all you would know is that they help you predict when you should become active.

Now let us consider the issue of how a region of cortex forms a name for a learned sequence. Again, imagine you are a region of cortex. Your active columns are changing with each new input. You have successfully learned the order in which your columns become active, meaning some of the cells in your columns become active prior to the arrival of inputs from lower regions. What information are you sending to the hierarchically higher regions of the cortex? We saw earlier that your layer 2 and layer 3 cells send their axons to the next higher regions. The activity of these cells is the input to the higher regions. But that’s a problem. In order for the hierarchy to work, you have to relay a constant pattern during learned sequences; you have to pass on the name of a sequence, not the details. Before you learn a sequence, you can pass on the details, but after you learn a sequence, and are able to successfully predict which columns will be active, you should relay only a constant pattern. However, I have not yet shown you a method for doing this. As it currently stands, you will pass on every changing pattern regardless of whether you can predict it. As each column becomes active, its layer 2 and layer 3 cells will send a new signal up the hierarchy. The cortex needs some way to keep the input to the next region constant during learned sequences. We need some way to turn off the output of the layer 2 and layer 3 cells when a column predicts its activity, or, alternately, to make these cells active when the column can’t predict its activity. This is the only way to make a constant name pattern.

There is not enough known about the cortex to state exactly how it does this. I can imagine several methods. I will describe my current favorite, but keep in mind that the concept is more important than the specific method. The creation of a constant “name” pattern is a requirement of this theory. All I can show at this time is that plausible mechanisms exist for the naming process.

Again imagine you are a column, as shown in figure 10. We want to understand how you learn to present a constant pattern to the next higher region when you can predict your activity, and a changing pattern when you can’t predict your activity. Let’s start by assuming that within layers 2 and 3 there are several classes of cells. (In addition to several types of inhibitory cells, many anatomists make the distinction between cell types in what they call layers 3a and 3b, so this assumption is not unreasonable.)

Let’s also assume that one class of cells, called layer 2 cells, learns to stay on during learned sequences. These cells, as a group, represent the name of the sequence. They will present a constant pattern to higher cortical regions as long as our region can predict what columns will be active next. If our region of cortex had a learned sequence of three different patterns, then the layer 2 cells of all the columns representing those three patterns would stay active as long as we were within that sequence. They are the name of the sequence.

Next, let’s assume there is another class of cells, layer 3b cells, which don’t fire when our column successfully predicts its input but do fire when it doesn’t predict its activity. A layer 3b cell represents an unexpected pattern. It fires when a column becomes active unexpectedly. It will fire every time a column becomes active prior to any learning. But as a column learns to predict its activity, the layer 3b cell becomes quiet. Together, the layer 2 and layer 3b cells meet our requirement. Before learning both cells fire on and off with the column, but after training the layer 2 cell is constantly active and the layer 3b cell is quiet.

How do these cells learn to do this? First, let’s consider how to shut down the layer 3b cell when its column successfully predicts its activity. Say that there is another cell positioned just above the layer 3b cell, in layer 3a. This cell also has dendrites in layer 1. Its only job is to prevent the layer 3b cell from firing when it sees the appropriate pattern in layer 1. When the layer 3a cell sees the learned pattern in layer 1, it quickly activates an inhibitory cell that prevents the layer 3b cell from firing. This is all it would take to stop the layer 3b cell from firing when the column correctly predicts its activity.

Now consider the more difficult task of keeping the layer 2 cell active throughout a known sequence of patterns. This is more difficult because a diverse set of layer 2 cells in many different columns would need to stay active together, even when their individual columns are not active. Here is how I believe this could occur. Layer 2 cells could learn to be driven purely from the hierarchically higher regions of cortex. They could form synapses preferentially with the axons from layer 6 cells in the regions above. The layer 2 cells would therefore represent the constant name pattern from the higher region. When a higher region of cortex sends a pattern down to layer 1 of the region below, a set of layer 2 cells in the lower region would become active, representing all the columns that are members of the sequence. Since these layer 2 cells also project back to the higher region, they would form a semistable group of cells. (It is unlikely that these cells stay constantly active. They probably fire synchronously in something like a rhythm.) It is as if the higher region sends the name of a melody to layer 1 below. This event causes a set of layer 2 cells to fire, one for each of the columns that will be active as the melody is heard.

The sum of all these mechanisms allows the cortex to learn sequences, make predictions, and form constant representations, or “names,” for sequences. These are the basic operations for forming invariant representations.

How do we make predictions about events we have never seen before? How do we decide among multiple interpretations of an input? How does a region of the cortex make specific predictions from invariant memories? I gave several examples of this earlier, such as predicting the exact next note in a melody when your memory only recalls the interval between the notes, the parable of the train, and reciting the Gettysburg Address. In these cases, the only way to solve this problem is to use the last specific information to convert an invariant prediction into a specific prediction. Another way to phrase this, in terms of the cortex, is to say that we have to combine feedforward information (actual input) with feedback information (a prediction in an invariant form).

Here is a simple example of how I believe this is done. Say your region of the cortex has been told to expect the musical interval of a fifth. The columns of your region represent all possible specific intervals such as C-E, C-G, D-A, etc. You need to decide which of your columns should be active. When the region above tells you to expect a fifth, it causes layer 2 cells to fire in all the columns that are fifths, such as C-G, D-A, and E-B. Cells in layer 2 of columns representing other intervals are not active. Now you have to select one column from all the possible fifths. The inputs to your region are specific notes. If the last note you heard was a D, then all the columns representing intervals involving a D, such as D-E and D-B, have partial input. So now in layer 2 we have activity in all columns that are fifths, and in layer 4 we have partial input to all columns representing intervals involving a D. The intersection of these two sets represents our answer, the column representing the interval D-A (see figure 11).

How does the cortex find this intersection? Recall that earlier I mentioned the fact that axons from layer 2 and layer 3 cells generally form synapses in layer 5 as they leave the cortex, and similarly axons approaching layer 4 from lower regions of cortex make a synapse in layer 6. The intersection of these two synapses (top down and bottom up) provides us with what is needed. A layer 6 cell that receives these two active inputs will fire. A layer 6 cell represents what a region of cortex believes is happening, a specific prediction. If a layer 6 cell could talk, it might say, “I am part of a column representing something. In my particular case my column represents the musical interval D-A. Other columns mean other things. I speak for my region of cortex. When I become active it means we believe the musical interval D-A either is occurring or will occur. I might become active because the bottom-up input from the ears caused the layer 4 cell in my column to excite the entire column. Or my activity might mean that we recognized a melody and are predicting this specific next interval. Either way, my job is to tell the lower regions of cortex what we think is happening. I represent

our interpretation of the world, regardless of whether it is true or just imagined.”

Let me describe this using another mental picture. Imagine two pieces of paper with lots of little holes punched in them. The holes on one paper represent the columns that have active layer 2 or layer 3 cells, our invariant prediction. The holes on the other paper represent columns with partial input from below. If you put one piece of paper on top of the other, some of the holes will line up, others won’t. The holes that line up represent the columns we think should be active.

This mechanism not only makes specific predictions, it also resolves ambiguities from the sensory inputs. Very often the input to a region of cortex will be ambiguous, as we saw with the colored papers, or when you hear a semi-garbled word. This bottom-up/top-down matching mechanism enables you to decide between two or more interpretations. And once you decide, you relay your interpretation to the region below.

Every moment in your waking life, each region of your neocortex is comparing a set of expected columns driven from above with the set of observed columns driven from below. Where the two sets intersect is what we perceive. If we had perfect input from below and perfect predictions, then the set of perceived columns would always be contained in the set of predicted columns. We often don’t have such agreement. The method of combining partial prediction with partial input resolves ambiguous input, it fills in missing pieces of information, and it decides between alternative views. It is how we combine an expected pitch-invariant interval with the last heard note to predict the next specific note in a melody. It is how we decide whether a picture is of a vase or of two faces. It is how we split our motor stream either to write or to speak the Gettysburg Address.

Finally, in addition to projecting to lower cortical regions, layer 6 cells can send their output back into layer 4 cells of their own column. When they do, our predictions become the input. This is what we do when daydreaming or thinking. It allows us to see the consequences of our own predictions. We do this many hours a day as we plan the future, rehearse speeches, and worry about events to come. Longtime cortical modeler Stephen Grossberg calls this “folded feedback.” I prefer “imagining.”

One last topic before we leave this section. I have pointed out several times that most often what we see, hear, or feel is highly dependent on our own actions. What we see is dependent on where our eyes saccade and how we turn our heads. What we feel is dependent on how we move our limbs and fingers. What we hear is sometimes dependent on what we say and do.

Therefore, to predict what we will sense next, we have to know what actions we are undertaking. Motor behavior and sensory perception are highly interdependent. How can we make predictions if what we sense next is largely a result of our own actions? Fortunately, there is a surprising and elegant solution to this problem, although many of the details are not understood.

The first surprising discovery is that perception and behavior are almost one and the same. As I mentioned earlier, most if not all regions of the cortex, even visual areas, participate in the creation of movement. The layer 5 cells that project to the thalamus and then to layer 1 also seem to have a motor function because they simultaneously project to motor areas of the old brain. Thus, the knowledge of “what just happened”—both sensory and motor—is available in layer 1.

The second surprising thing, and a consequence of the first, is that motor behavior must also be represented in a hierarchy of invariant representations. You generate the movements necessary to carry out a particular action by thinking of doing it in a detail-invariant form. As the motor command travels down the hierarchy, it gets translated into the complex and detailed sequences required to perform the activity you expected to do. This is happening in both “motor” cortex and “sensory” cortex, which blurs the distinction between the two. If region IT of visual cortex is perceiving “nose,” the mere act of switching to the representation for “eye” will generate the saccade necessary to make this prediction a reality. The particular saccade necessary to move from seeing a nose to seeing an eye varies depending on where the face is. A close face requires a larger saccade; a more distant face requires a smaller saccade. A tilted face requires saccading at an angle different from the one for a level face. The details of the needed saccade are determined as the prediction of seeing the “eye” moves toward V1. The saccade becomes increasingly specific the farther down it goes, resulting in a saccade that lands your foveas right on target, or pretty close.

Let’s look at another example. For me to physically move from my living room to my kitchen, all my brain has to do is mentally switch from the invariant representation of my living room to the invariant representation of my kitchen. This switch causes a complex unfolding of sequences. The process of generating the sequence of predictions of what I will see, feel, and hear while walking from the living room to the kitchen also generates the sequence of motor commands that makes me walk from my living room to my kitchen and move my eyes as I do so. Prediction and motor behavior work hand in hand as patterns flow down and up the cortical hierarchy. As strange as it sounds, when your own behavior is involved, your predictions not only precede sensation, they determine sensation. Thinking of going to the next pattern in a sequence causes a cascading prediction of what you should experience next. As the cascading prediction unfolds, it generates the motor commands necessary to fulfill the prediction. Thinking, predicting, and doing are all part of the same unfolding of sequences moving down the cortical hierarchy.

“Doing” by thinking, the parallel unfolding of perception and motor behavior, is the essence of what is called goal-oriented behavior. Goal-oriented behavior is the holy grail of robotics. It is built into the fabric of the cortex.

We can turn off our motor behavior, of course. I can think of seeing something without actually seeing it and I can think of going to my kitchen without actually doing so. But thinking of doing something is literally the start of how we do it.

Let’s step back a bit and think some more about how information moves up and down the cortical hierarchy. As you move about the world, changing inputs stream into lower regions of the cortex. Each region tries to interpret its stream of inputs as part of a known sequence of patterns. The columns try to anticipate their activity. If they can, they will pass on a stable pattern, the name of the sequence, to the next higher region. Again it is as if the region says, “I am listening to a song, here is the name. I can handle the details.”

But what if an unexpected pattern arrives, an unexpected note? Or what if we see something that does not belong on a face? The unexpected pattern is automatically passed to the next higher cortical region. This happens naturally as layer 3b cells that were not part of the expected sequence fire. The higher region may be able to understand this new pattern as the next part of its own sequence. It might say, “Oh, I see a new note arrived. Maybe this is the first note of the next song on the album. It looks like it, so I predict we have gone on to the next song. Lower region, here is the name of the next song I think you should be hearing.” But if such recognition does not occur, an unexpected pattern will keep propagating up the cortical hierarchy until some higher region can interpret it as part of its normal sequence of events. The higher the unexpected pattern needs to go, the more regions of the cortex get involved in resolving the unexpected input. Finally, when a region somewhere up the hierarchy thinks it can understand the unexpected event, it generates a new prediction. This new prediction propagates down the hierarchy as far as it can go. If the new prediction is not right, an error will be detected, and again it will climb up the hierarchy until some region can interpret it as part of its currently active sequence. Thus we can see that observed patterns flow up the hierarchy and predictions flow down the hierarchy. Ideally, in a world that is known and predictable, most of the up-and-down flow of patterns happens rapidly and occurs in the lower regions of the cortex. The brain quickly tries to find a part of its world model that is consistent with any unexpected input. Only then will it understand its input and know what to expect next.

If I am walking in a familiar room in my house, few errors will propagate high up my cortex. The highly learned sequences of my house can be handled in the lower sections of the visual, somatosensory, and motor hierarchy. I know the room so well I can even walk around in the dark. My familiarity with the surroundings effectively frees up most of my cortex for other tasks such as thinking about brains and writing books. However, if I was in an unknown room, especially one that was different from any I had seen before, not only would I need to look to see where I was going, but unexpected patterns would constantly be rising far up the cortical hierarchy. The more my sensory experience doesn’t match learned sequences, the more the errors rise up. In this novel situation, I can no longer think about brains because most of my cortex is attending to the problems of navigating the room. This is a common experience for people who step off an airplane into a foreign country. While the roads may look like those you’re accustomed to, cars may whiz down the wrong side, the money is strange, the language is incomprehensible, and learning how to find a bathroom can take all your cortical powers. Don’t try to rehearse a speech while walking in a foreign land.

The sensation of sudden comprehension, the “aha!” moment, can be understood in this model. Imagine you are looking at an ambiguous picture. Filled with blobs of ink and scattered lines, it doesn’t look like anything. It doesn’t make sense. Confusion occurs when the cortex can’t find any memory that matches with the input. Your eyes scan everywhere on the picture. New inputs race all the way up the cortical hierarchy. High-level cortex tries lots of different hypotheses but, as these predictions race down the hierarchy, each and every one conflicts with the input and the cortex is forced to try again. During this time of confusion your brain is totally occupied with understanding the picture. Finally, you make a high-level prediction

that is the right one. When this happens, the prediction starts at the top of the cortical hierarchy and succeeds in propagating all the way to the bottom, chunk, chunk, chunk, chunk, chunk. In less than a second, each region is given a sequence that fits the data. No more errors rise to the top. You understand the picture, you see a dalmatian dog amid the dots and scribbles (see figure 12).

We’ve known for decades that connections in the cortical hierarchy are reciprocal. If region A projects to region B, then B projects to region A. There are often more axon fibers going backward than forward. But even though this description is widely accepted, the prevailing paradigm is that feedback plays a minor or “modulatory” role in the brain. The idea that a feedback signal could instantly and accurately cause a diverse set of cells in layer 2 to fire is not the prevailing view among neuroscientists.

Why would this be so? Part of the reason, as I mentioned before, is that there’s no real need to be concerned with feedback if you don’t accept the central role of prediction. If you think that the information flows straight through to the motor system, then why do you need feedback? Another reason to ignore feedback is that the feedback signal is spread over large areas of layer 1. Intuitively, we would expect a signal dispersed over a large area would have only a minor effect on many neurons, and indeed the brain has several such modulatory signals that don’t act on specific neurons but change global attributes such as alertness.

The final reason to ignore feedback is due to how many scientists believe individual neurons work. Typical neurons have thousands or tens of thousands of synapses. Some are located far from the cell body, others are right up close to it. Synapses close to the cell body have a strong influence on cell firing. A dozen or so active synapses near the cell body can cause it to generate a spike or pulse of electrical discharge. This is known. However, the vast majority of synapses are not close to the body of the cell. They are spread far and wide out over the branchlike structure of a cell’s dendrites. Since these synapses are far away from the cell’s body, scientists have tended to believe that a spike arriving at one of those synapses would have a weak or almost imperceptible effect on whether the neuron generates a spike. The effect of a distant synapse would dissipate by the time it reaches the cell body.

As a general rule, information flowing up the cortical hierarchy is transferred via synapses close to cell bodies. Information going up the hierarchy is therefore more certain to pass from region to region. Also, as a general rule, feedback flowing down the cortical hierarchy does so via synapses far from the cell body. Cells in layers 2, 3, and 5 send dendrites into layer 1 and form many synapses there. Layer 1 is a mass of synapses, but they are all far from the cell bodies in layers 2,3, and 5. Furthermore, any particular cell in layer 2, say, will only form a few, if any, synapses with any particular feedback fiber. Therefore, some scientists may object to the idea that a brief pattern in layer 1 could accurately cause a set of cells to fire in layers 2, 3, and 5. But that is precisely what the theory I have laid out requires.

The resolution to this dilemma is that neurons behave differently from the way they do in the classic model. In fact, in recent years there has been a growing group of scientists who have proposed that synapses on distant, thin dendrites can play an active and highly specific role in cell firing. In these models, these distant synapses behave differently from synapses on thicker dendrites near the cell body. For example, if there were two synapses very close to each other on a thin dendrite, they would act as a “coincidence detector.” That is, if both synapses received an input spike within a small window of time, they could exert a large effect on the cell even though they are far from the cell body. They could cause the cell body to generate a spike. How the dendrites of a neuron behave is still a mystery, so I cannot say much more about them here. What is important is that the memory-prediction model of cortex requires that synapses far from the cell body be able to detect specific patterns.

In hindsight, it seems almost foolish to say that most of the thousands of synapses on a neuron play only a modulatory role. Massive feedback and massive numbers of synapses exist for a reason. Using this insight, we can say that a typical neuron has the ability to learn hundreds of precise coincidences on feedback fibers as they make synapses on thin dendrites. It means that each column in our neocortex is highly flexible in terms of which feedback patterns cause it to become active. It means that any particular feature can be precisely associated with thousands of different objects and sequences. My model requires feedback to be fast and precise. Cells need to fire when they see any number of precise coincidences on their distant dendrites. These new neuron models allow for this.

All the cells in all the layers of the cortex have synapses, and most of these synapses can be modified via experience. It is safe to say that learning and memory occur in all layers, in all columns, in all regions of cortex.

Earlier in the book I mentioned Hebbian learning, named after the Canadian neuropsychologist Donald O. Hebb. Its essence is very simple: When two neurons fire at the same time, the synapses between them get strengthened. (It is easily summarized in the phrase “Fire together, wire together.”) We now know Hebb was basically right. Of course, nothing in nature is ever quite so simple, and in real brains the details are more complicated. Our nervous systems run many variations of Hebb’s learning rule; for instance, some synapses change their strength in response to small variations in the timing of neural signals, some synaptic changes are short-lived, and some changes are long-lived. But Hebb was erecting only a framework for the study of learning, not a final theory, and that framework has been incredibly useful.

Hebbian learning principles can explain most of the cortical behavior I have mentioned in this chapter. Remember, it was shown back in the 1970s that auto-associative memories using the classical Hebbian learning algorithm can learn spatial patterns and sequences of patterns. The main problem was that the memories couldn’t handle variation well. According to the theory proposed in this book, the cortex has gotten around this limitation partly by stacking auto-associative memories in a hierarchy and partly by using a sophisticated columnar architecture. This chapter has been almost all about the hierarchy and how it works, because the hierarchy is what makes the cortex powerful. So instead of going through in painful detail how each cell could learn this or that, I want to cover a few broad principles of learning in a hierarchy.

When you are born your cortex essentially doesn’t know anything. It doesn’t know about your language, your culture, your home, your town, songs, the people you will grow up with, nothing. All this information, the structure of the world, has to be learned. The two basic components of learning are forming the classifications of patterns and building sequences. These two complementary memory components interact. As one region learns sequences, the inputs it sends to the layer 4 cells in higher cortical regions change. These layer 4 cells therefore learn to form new classifications, which changes the pattern projected back to layer 1 in the lower region, which affects the sequences.

The basics of forming sequences is to group patterns together that are part of the same object. One way to do this is by grouping patterns that occur contiguously in time. If a child holds a toy in her hand and slowly moves it, her brain can safely assume that the image on her retina is of the same object moment to moment, and therefore the changing set of patterns can be grouped together. At other times you need outside instruction to help you decide which patterns belong together. To learn that apples and bananas are fruits, but carrots and celery are not, requires a teacher to guide you to group these items as fruits. Either way, your brain slowly builds sequences of patterns that belong together. But as a region of cortex builds sequences, the input to the next region changes. The input changes from representing mostly individual patterns to representing groups of patterns. The input to a region changes from notes to melodies, from letters to words, from noses to faces, and so on. Because the bottom-up inputs to a region become more “object-oriented,” the higher region of cortex can now learn sequences of these higher-order objects. Where before a region built sequences of letters, it now builds sequences of words. The unexpected result of this learning process is that, during repetitive learning, representations of objects move down the cortical hierarchy. During the early years of your life, your memories of the world first form in higher regions of cortex, but as you learn they are re-formed in lower and lower parts of the cortical hierarchy. It isn’t that the brain moves them; it has to relearn them over and over. (I am not suggesting that all memories start at the top of the cortex. The actual formation of memories is more complex. I believe layer 4 pattern classification starts at the bottom and moves up. But as it does, we start forming sequences that then move down. It is the memory of sequences I am suggesting re-form lower and lower in the cortex.) As simple representations move down, the regions at the top are able to learn more complex and subtle patterns.

You can observe the creation and downward movement of hierarchical memory by observing how a child learns. Consider how we learn to read. The first thing we learn is to recognize individual printed letters. This is a slow and difficult task requiring conscious effort. Then we move on to recognizing simple words. Again, it is difficult and slow at first, even for three-letter words. The child can read each letter in sequence and sound out the letters one after another, but it takes a fair amount of practice before the word itself is recognized as a word. After learning simple words, we struggle with complex, multisyllable words. At first, we sound out each syllable, concatenating them as we did with letters when learning simple words. After years of practice, a person can read quickly. We get to the point where we don’t actually see all the individual letters but instead recognize entire words and often entire phrases at a glance. It isn’t just that we are faster; we are actually recognizing words and phrases as entities. When we read an entire word at one time, do we still see the letters? Yes and no. Obviously, the retina sees the letters and therefore so do regions of V1. But the recognition of the letters is occurring fairly low in the cortical hierarchy, say in V2 or V4. By the time the signal gets to IT, the individual letters are no longer represented. What at first took the effort of your entire visual cortex—recognizing individual letters—is now occurring closer to the sensory input. As memory of simple objects like letters moves down the hierarchy, the higher regions have the ability to learn complex objects like words and phrases.

Learning to read music is another example. At first you have to concentrate on every note. With practice, you start to recognize common note sequences, then entire phrases. After much practice, it is as if you don’t see most of the notes at all. The sheet music is there only to remind you of the major structure of the piece; the detailed sequences have been memorized lower down. This type of learning occurs in both motor and sensory areas.

A young brain is slower to recognize inputs and slower to make motor commands because the memories used in these tasks are higher up the cortical hierarchy. Information has to flow all the way up and down, maybe with multiple passes, to resolve conflicts. It takes time for the neural signals to travel up and down the cortical hierarchy. A young brain also has not yet formed complex sequences at the top and therefore cannot recognize and play back complex patterns. A young brain cannot understand the higher-order structure of the world. Compared to an adult’s, a child’s language is simple, his music is simple, and his social interactions are simple.

If you study a particular set of objects over and over, your cortex re-forms memory representations for those objects down the hierarchy. This frees up the top for learning more subtle, more complex relationships. According to the theory, this is what makes an expert.

In my work designing computers, some people are surprised by how quickly I can look at a product and see the problems inherent in its design. After twenty-five years of designing computers, I have a better-than-average model of the issues associated with mobile computing devices. Similarly, an experienced parent can easily recognize why his child is upset, whereas a first-time parent may struggle with how to handle a situation. An experienced business manager can readily see the flaws and advantages of the structure of an organization whereas the novice manager just can’t understand these things. They have the same input, but the novice’s model is not as sophisticated. In all such cases and a thousand more, we start by learning the basics, the simplest structure. Over time we move our knowledge down the cortical hierarchy and, therefore, we have the opportunity at the top for learning higher-order structure. It is this higher-order structure that makes us experienced. Experts and geniuses have brains that see structure of structure and patterns of patterns beyond what others do. You can become expert by practice, but there certainly is a genetic component to talent and genius too.

Three large brain structures lie under the neocortical sheet and communicate with it. They are the basal ganglia, the cerebellum, and the hippocampus. All three existed prior to the neocortex. With a very broad brush, we can say that the basal ganglia were the primitive motor system, the cerebellum learned precise timing relationships of events, and the hippocampus stored memories of specific events and places. To some extent, the neocortex has subsumed their original functions. For example, a human born without much of a cerebellum will have deficits in timing and will have to apply more conscious effort when moving but otherwise will be pretty normal.

We know that the neocortex is responsible for all complex motor sequences and can directly control your limbs. It isn’t that the basal ganglia are unimportant, just that the neocortex has taken over a large part of motor control. Because of this, I have described the overall function of the neocortex independently of the basal ganglia and the cerebellum. Some scientists may disagree with this assumption, but it is one I have used in this book and in my work.

The hippocampus, however, is a different beast. It is one of the most heavily studied areas in the brain because it is essential to the formation of new memories. If you lose both halves of the hippocampus (like many parts of the nervous system, it exists in both the left and the right sides of the brain), you lose the ability to form most new memories. Without the hippocampus you can still talk, walk, see, and hear, and for brief periods of time appear almost normal. But in fact you are profoundly impaired: you can’t remember anything new. You can remember a friend you knew before you lost the hippocampus, but you cannot remember a new person. Even if you met with your doctor five times a day for a year, each time would be like the first time. You would have no memory of events that occurred after the loss of your hippocampus.

For many years I hated to think about the hippocampus because it didn’t make sense to me. It clearly is essential for learning, yet it isn’t the ultimate repository for most of what we know. The neocortex is. The classic view of the hippocampus is that new memories are formed there, and later, over a period of days, weeks, or months, these new memories are transferred to the neocortex. This made no sense to me. We know that sights, sound, touch—our sensory data stream—flow directly into the sensory areas of the cortex without first passing through the hippocampus. It seemed to me that this sensory information should automatically form new memories in the cortex. Why do we need a hippocampus to learn? How could a separate structure such as the hippocampus interfere and prevent learning in the cortex, only later to transfer the information back to the cortex?

I decided to put the hippocampus aside, figuring that some day its role would become clear. That day happened in late 2002, right around the time I began writing this book. One of my colleagues at the Redwood Neuroscience Institute, Bruno Olshausen, pointed out that the connections between the hippocampus and the neocortex suggest that the hippocampus is the top region of the neocortex, not a separate structure. In this view, the hippocampus occupies the peak of the neocortical pyramid, the top block in figure 5. The neocortex appeared on the evolutionary scene sandwiched between the hippocampus and the rest of the brain. Apparently this view of the hippocampus as being at the top of the cortical hierarchy was known for some time; I was just not aware of it. I had spoken to several hippocampus experts and asked them to explain how this seahorse-shaped structure could transfer memories to the cortex. No one could. And no one mentioned that the hippocampus was at the top of the cortical pyramid, probably because the hippocampus not only sits at the top of the cortical pyramid, but it still connects directly to many older parts of the brain.

Yet I instantly saw this new perspective as the solution to my confusion.

Think about information flowing from your eyes, ears, and skin into the neocortex. Each region of the neocortex tries to understand what this information means. Each region tries to understand the input in terms of the sequences it knows. If it does understand the input, it says, “I understand this, it is just part of the object I am already seeing. I won’t pass on the details.” If a region doesn’t understand the current input, it passes it up the hierarchy until some higher region does. However, a pattern that is truly novel will escalate further and further up the hierarchy. Each successively higher region says, “I don’t know what this is, I didn’t anticipate it, why don’t you higherups look at it?” The net effect is that when you get to the top of the cortical pyramid, what you have left is information that can’t be understood by previous experience. You are left with the part of the input that is truly new and unexpected.

In a typical day we encounter many new things that make it to the top—for example, a story in the newspaper, the name of the person you met this morning, and the car accident you saw on the way home. It is these unexplained and unanticipated remainders, the new stuff, that enter the hippocampus and are stored there. This information won’t be stored forever. Either it will be transferred down into the cortex below or it will eventually be lost.

I have noticed that, as I get older, I have trouble remembering new things. For example, my children remember the details of most of the theatrical plays they have seen in the last year. I can’t. Perhaps it is because I have seen so many plays in my life that rarely do I see anything truly new. New plays fit into memories of past plays, and the information just doesn’t make it to my hippocampus. For my children, each play is more novel and does reach the hippocampus. If this is true, we could say the more you know, the less you remember.

Unlike the neocortex, the hippocampus has a heterogeneous structure with several specialized regions. It’s good at the unique task of quickly storing whatever patterns it sees. The hippocampus is in the perfect position, at the top of the cortical pyramid, to remember what is novel. It is also in the perfect position to recall these novel memories, allowing them to be stored in the cortical hierarchy, which is a somewhat slow process. You can instantly remember a novel event in the hippocampus, but you will permanently remember something in the cortex only if you experience it over and over, either in reality or by thinking of it.

Your cortex has a second major pathway for passing information from region to region, up the hierarchy. This alternate path starts with cells in layer 5, which project to the thalamus (a different part of the thalamus from the one we discussed earlier) and then from the thalamus up to the next higher region of cortex. Whenever two regions of cortex directly connect to each other in a hierarchical fashion, they also connect indirectly through the thalamus. This second pathway passes information only up the hierarchy, not down. So as we move up the cortical hierarchy, there is a direct path between two regions and an indirect path through the thalamus.

The second path has two modes of operation determined by cells in the thalamus. In one mode, the path is mostly shut down so information does not flow through. In the other mode, information flows accurately between regions. Two scientists, Murray Sherman of the State University of New York at Stony Brook and Ray Guillery of the University of Wisconsin School of Medicine, have described this alternate pathway and postulated that it might be as important as the direct one (perhaps more so), which has been the subject of this chapter so far. I have a speculation of what this second pathway is doing.

Read this word: imagination. Most people can recognize the word in a single glance, one fixation. Now look at the letter i in the middle of the word. Now look at the dot over that i. Your eyes can be fixated on the exact same location, but in one case you see the word, in the next case you see the letter, and in the final case you see the dot. Stare at the i and try to alternate your perception between the word, the letter, and the dot. If you have trouble, try saying “dot,” “i,” and “imagination” while you stare at the dot. In all cases the exact same information is entering VI, yet by the time it gets to some higher region like IT you perceive different things, different levels of detail. Region IT knows how to recognize all three objects. It can recognize the dot in isolation, the letter i, and the entire word at a glance. But when you perceive the entire word, V4, V2, and V1 handle the details, and all IT gets to know about is the word. You normally don’t perceive the individual letters while reading; you perceive words or phrases. But you can perceive the letters if you choose to. We are doing this sort of attentional shift all the time, but we aren’t generally aware of it. I can be listening to music playing in the background and barely be aware of the melody, but if I try, I can isolate the singer or the bass guitar. The same sound is entering my head, but I can focus my perceptions. Every time you scratch your head, the movement makes a loud internal sound, but usually you are unaware of the noise. If you focus on it, though, you can hear the sound clearly. This is another example of sensory input that normally is handled low in the cortical hierarchy but can be brought to higher levels if you attend to it.

I speculate that the alternate pathway through the thalamus is the mechanism by which we attend to details that normally we wouldn’t notice. It bypasses the grouping of sequences in layer 2, sending the raw data to the next higher region of cortex. Biologists have shown that the alternate pathway can be turned on in one of two ways. One is by a signal from the higher region of the cortex itself. This is the method you used when I asked you to attend to details that normally you wouldn’t notice, such as the dot on the i or the sound of a head scratch. The second method that can activate this pathway is a large, unexpected signal from below. If the input to the alternate pathway is strong enough, it sends a wake-up signal to the higher region, which again can turn on the pathway. For example, if I showed you a face and asked you what it was, you would say “Face.” If I showed you the same face but it had a strange mark on the nose, you would first recognize the face, but then immediately your lower levels of vision would notice that something is wrong. This error forces the opening of the attentional pathway. The details will now take the alternate path, bypassing the grouping that normally occurs, and your attention will be drawn to the mark. You now see the mark and not just the face. If it was sufficiently odd, the mark could occupy your entire attention. In this way, unusual events quickly rise to your attention. This is why we can’t avoid focusing on deformities and other unusual patterns. Your brain does this automatically. Often, however, errors aren’t strong enough to open the alternate pathway. This is why we sometimes don’t notice if a word is misspelled as we read it.

To find and establish a new scientific framework, it is necessary to look for the simplest concepts capable of uniting and explaining what were large quantities of disparate facts. It is an unavoidable consequence of this process that the pendulum swings too far toward simplification. Important details are likely to be ignored, and facts will be misinterpreted. If the framework takes hold, refinements and fixes will inevitably be found showing where the initial proposal went too far, didn’t go far enough, or was in error.

In this chapter, I have introduced many speculative ideas on how the neocortex works. I expect that several of these ideas will prove to be wrong, and probably all of these ideas will be revised. There are also many details I haven’t even mentioned. The brain is very complex; neuroscientists reading this book will know that I have presented a crude characterization of the complexities of the real brain. Yet I believe the framework as a whole to be sound. All I can hope is that the core ideas will be preserved as the details change in the face of new data and understanding.

Finally, you may be struggling with the idea that a simple but large memory system could actually result in all that humans do. Could you and I really be just a hierarchical memory system? Could our lives, beliefs, and ambitions be stored in trillions of tiny synapses? In 1984, I started writing computer programs professionally. I had written small programs before but this was my first time programming a computer with a graphical user interface, and my first time working on large and sophisticated applications. I was writing software for an operating system created by Grid Systems. With windows, multiple fonts, and menus, the Grid operating system was quite advanced for its time.

One day I was struck by the near impossibility of what I was doing. As a programmer, I wrote one line of code at a time. I grouped lines of code into blocks called subroutines. Subroutines were grouped into modules. Modules were combined to form an application. The spreadsheet program I was working on had so many subroutines and modules that no one person could understand the entire thing. It was complex. Yet a single line of code would do very little. To place one pixel on the display took several lines of code. To draw an entire screen for the spreadsheet required the computer to execute millions of instructions that were spread over hundreds of subroutines. Subroutines invoked other subroutines in repetitive and recursive ways. It was so complicated that it wasn’t possible to know everything that would happen when the program ran. It struck me as highly improbable that when the program did run it would draw its image in what seemed like an instant. Its outward appearance was tables of numbers, labels, text, and graphs. It behaved like a spreadsheet. But I knew what was going on inside the computer with the processor executing one simple instruction at a time. It was hard to believe that the computer could find its way through the maze of modules and subroutines and execute all those instructions so quickly. If I hadn’t known better, I would have been certain the whole thing couldn’t work. I realized that if someone had invented the concept of a computer with a graphical user interface and a spreadsheet application, and presented it to me on paper, I would have rejected it as impractical. I would have said it would take forever to do anything. It was a humbling thought because it did work. It was then that I realized my intuitive sense for the speed of the microprocessor and my intuitive sense for the power of hierarchical design were inadequate.

There is a lesson here about the neocortex. It isn’t made of superfast components and the rules under which it operates are not that complex. However, it does have a hierarchical structure that contains billions of neurons and trillions of synapses. If we find it hard to imagine how such a logically simple but numerically vast memory system can create our consciousness, our languages, our cultures, our art, this book, and our science and technology, I suggest it is because our intuitive sense of the capacity of the cortex and the power of its hierarchical structure is inadequate. The neocortex does work. It isn’t magic. We can understand it. And like a computer, ultimately we can build intelligent machines that work on the same principles.