One day in April 1986 I was contemplating what it means to “understand” something. For months I had been struggling with the fundamental question What do brains do if they aren’t generating behavior? What does a brain do when it is passively listening to speech? What is your brain doing right now while it is reading? Information goes into the brain but doesn’t come out. What happens to it? Your behaviors at the moment are probably basic—such as breathing and eye movements—yet, as you are aware, your brain is doing a lot more than that as you read and understand these words. Understanding must be the result of neural activity. But what? What are the neurons doing when they understand?

As I looked around my office that day, I saw familiar chairs, posters, windows, plants, pencils, and so on. There were hundreds of items and features all around me. My eyes saw them as I glanced around, yet just seeing them didn’t cause me to perform any action. No behavior was invoked or required, yet somehow I “understood” the room and its contents. I was doing what Searle’s Chinese Room couldn’t do, and I didn’t have to pass anything back through a slot. I understood, but had no action to prove it. What did it mean to “understand”?

It was while pondering this dilemma that I had an “aha” insight, one of those emotionally powerful moments when suddenly what was a tangle of confusion becomes clear and understood. All I did was ask what would happen if a new object, one I had never seen before, appeared in the room—say, a blue coffee cup.

The answer seemed simple. I would notice the new object as not belonging. It would catch my attention as being new. I needn’t consciously ask myself if the coffee cup was new. It would just jump out as not belonging. Underlying that seemingly trivial answer is a powerful concept. To notice that something is different, some neurons in my brain that weren’t active before would have to become active. How would these neurons know that the blue coffee cup was new and the hundreds of other objects in the room were not? The answer to this question still surprises me. Our brains use stored memories to constantly make predictions about everything we see, feel, and hear. When I look around the room, my brain is using memories to form predictions about what it expects to experience before I experience it. The vast majority of predictions occur outside of awareness. It’s as if different parts of my brain were saying, “Is the computer in the middle of the desk? Yes. Is it black? Yes. Is the lamp in the right-hand corner of the desk? Yes. Is the dictionary where I left it? Yes. Is the window rectangular and the walls vertical? Yes. Is sunlight coming from the correct direction for the time of day? Yes.” But when some visual pattern comes in that I had not memorized in that context, a prediction is violated. And my attention is drawn to the error.

Of course, the brain doesn’t talk to itself while making predictions, and it doesn’t make predictions in a serial fashion. It also doesn’t just make predictions about distinct objects like coffee cups. Your brain constantly makes predictions about the very fabric of the world we live in, and it does so in a parallel fashion. It will just as readily detect an odd texture, a misshapen nose, or an unusual motion. It isn’t immediately apparent how pervasive these mostly unconscious predictions are, which is perhaps why we missed their importance for so long. They happen so automatically, so easily, we fail to fathom what is happening inside our skulls. I hope to impress on you the power of this idea. Prediction is so pervasive that what we “perceive”—that is, how the world appears to us—does not come solely from our senses. What we perceive is a combination of what we sense and of our brains’ memory-derived predictions.

Minutes later I conceived a thought experiment to help convey what I understood at that moment. I call it the altered door experiment. Here is how it goes.

When you come home each day, you usually take a few seconds to go through your front door or whichever door you use. You reach out, turn the knob, walk in, and shut it behind you. It’s a firmly established habit, something you do all the time and pay little attention to. Suppose while you are out, I sneak over to your home and change something about your door. It could be almost anything. I could move the knob over by an inch, change a round knob into a thumb latch, or turn it from brass to chrome. I could change the door’s weight, substituting solid oak for a hollow door, or vice versa. I could make the hinges squeaky and stiff, or make them glide frictionlessly. I could widen or narrow the door and its frame. I could change its color, add a knocker where the peephole used to be, or add a window. I can imagine a thousand changes that could be made to your door, unbeknownst to you. When you come home that day and attempt to open the door, you will quickly detect that something is wrong. It might take you a few seconds’ reflection to realize exactly what is wrong, but you will notice the change very quickly. As your hand reaches for the moved knob, you will realize that it is not in the correct location. Or when you see the door’s new window, something will appear odd. Or if the door’s weight has been changed, you will push with the wrong amount of force and be surprised. The point is that you will notice any of a thousand changes in a very short period of time.

How do you do that? How do you notice these changes? The AI or computer engineer’s approach to this problem would be to create a list of all the door’s properties and put them in a database, with fields for every attribute a door can have and specific entries for your particular door. When you approach the door, the computer would query the entire database, looking at width, color, size, knob position, weight, sound, and so on. While this may sound superficially similar to how I described my brain checking each of its myriad predictions as I glanced around my office, the difference is real and far-reaching. The AI strategy is implausible. First, it is impossible to specify in advance every attribute a door can have. The list is potentially endless. Second, we would need to have similar lists for every object we encounter every second of our lives. Third, nothing we know about brains and neurons suggests that this is how they work. And finally, neurons are just too slow to implement computer-style databases. It would take you twenty minutes instead of two seconds to notice the change as you go through the door.

There is only one way to interpret your reaction to the altered door: your brain makes low-level sensory predictions about what it expects to see, hear, and feel at every given moment, and it does so in parallel. All regions of your neocortex are simultaneously trying to predict what their next experience will be. Visual areas make predictions about edges, shapes, objects, locations, and motions. Auditory areas make predictions about tones, direction to source, and patterns of sound.

Somatosensory areas make predictions about touch, texture, contour, and temperature.

“Prediction” means that the neurons involved in sensing your door become active in advance of them actually receiving sensory input. When the sensory input does arrive, it is compared with what was expected. As you approach the door, your cortex is forming a slew of predictions based on past experience. As you reach out, it predicts what you will feel on your fingers, when you will feel the door, and at what angle your joints will be when you actually touch the door. As you start to push the door open, your cortex predicts how much resistance the door will offer and how it will sound. When your predictions are all met, you’ll walk through the door without consciously knowing these predictions were verified. But if your expectations about the door are violated, the error will cause you to take notice. Correct predictions result in understanding. The door is normal. Incorrect predictions result in confusion and prompt you to pay attention. The door latch is not where it’s supposed to be. The door is too light. The door is off center. The texture of the knob is wrong. We are making continuous low-level predictions in parallel across all our senses.

But that’s not all. I am arguing a much stronger proposition. Prediction is not just one of the things your brain does. It is the primary function of the neocortex, and the foundation of intelligence. The cortex is an organ of prediction. If we want to understand what intelligence is, what creativity is, how your brain works, and how to build intelligent machines, we must understand the nature of these predictions and how the cortex makes them. Even behavior is best understood as a by-product of prediction.

I don’t know who was the first person to suggest that prediction is key to understanding intelligence. In science and industry no one invents anything completely new. Rather, people see how existing ideas fit into new frameworks. The components of a new idea are usually floating around in the milieu of scientific discourse prior to its discovery. What is usually new is the packaging of these components into a cohesive whole. Similarly, the idea that a primary function of the cortex is to make predictions is not entirely new. It has been floating around in various forms for some time. But it has not yet assumed its rightful position at the center of brain theory and the definition of intelligence.

Ironically, some of the pioneers of artificial intelligence had a notion of computers building a model of the world and using it to make predictions. In 1956, for example, D. M. Mackay argued that intelligent machines should have an “internal response mechanism” designed to “match what is received.” He didn’t use the words “memory” and “prediction” but he was thinking along the same lines.

Since the mid-1990s, terms such as inference, generative models, and prediction have crept into the scientific nomenclature. They all refer to related ideas. As an example, in his 2001 book, i of the vortex, Rodolfo Llinas, at the New York University School of Medicine, wrote, “The capacity to predict the outcome of future events—critical to successful movement—is, most likely, the ultimate and most common of all global brain functions.” Scientists such as David Mumford at Brown University, Rajesh Rao at the University of Washington, Stephen Grossberg at Boston University, and many more have written and theorized about the role of feedback and prediction in various ways. There is an entire subfield of mathematics devoted to Bayesian networks. Named after Thomas Bayes, an English minister born in 1702 who was a pioneer in statistics, Bayesian networks use probability theory to make predictions.

What has been lacking is putting these disparate bits and pieces into a coherent theoretical framework. This, I argue, has not been done before, and it is the goal of this book.

Before we get into detail about how the cortex makes predictions, let’s consider some additional examples. The more you think about this idea, the more you’ll realize that prediction is pervasive and the basis for how you understand the world.

This morning I made pancakes. At one point in the process, I reached under the counter to open a cabinet door. I intuitively knew, without seeing, what I would feel—in this case, the cabinet doorknob—and when I would feel it. I twisted the top of the milk container with the expectation that it would turn and then come free. I turned on the griddle expecting the knob to push in a slight amount, then turn with a certain resistance. I expected to hear the gentle fwoomp of the gas flame about a second later. Every minute in the kitchen I made dozens or hundreds of motions, and each one involved many predictions. I know this because if any of those common motions had had a different result from the expected one, I would have noticed it.

Every time you put your foot down while you are walking, your brain predicts when your foot will stop moving and how much “give” the material you step on will have. If you have ever missed a step on a flight of stairs, you know how quickly you realize something is wrong. You lower your foot and the moment it “passes through” the anticipated stair tread you know you are in trouble. The foot doesn’t feel anything, but your brain made a prediction and the prediction was not met. A computer-driven robot would blissfully fall over, not realizing that anything was amiss, while you would know as soon as your foot continues for even a fraction of an inch beyond the spot where your brain had expected it to stop.

When you listen to a familiar melody, you hear the next note in your head before it occurs. When you listen to a favorite album, you hear the beginning of each next song a couple of seconds before it starts. What’s happening? Neurons in your brain that will fire when you hear that next note fire in advance of your actually hearing it, and so you “hear” the song in your head. The neurons fire in response to memory. This memory can be surprisingly long lasting. It is not uncommon to listen to an album of music for the first time in many years and still hear the next song automatically after the previous song has ended. And it creates a pleasant sensation of mild uncertainty when you listen to your favorite CD on random shuffle; you know your prediction of the next song is wrong.

When listening to people speak, you often know what they’re going to say before they’ve finished speaking—or at least you think you know! Sometimes we don’t even listen to what the speaker actually says and instead hear what we expect to hear. (This happened to me so often when I was a child that my mother twice took me to a doctor to have my hearing checked.) You experience this in part because people tend to use common phrases or expressions in much of their conversation. If I say, “How now brown …,” your brain will activate neurons that represent the word cow before I say it (though if English is not your native language, you may have no idea what I am talking about). Of course, we don’t know all the time what others are going to say. Prediction is not always exact. Rather, our minds work by making probabilistic predictions concerning what is about to happen. Sometimes we know exactly what is going to happen, other times our expectations are distributed among several possibilities. If we were eating at a table in a diner and I said, “please pass me the …,” your brain would not be surprised if I next said, “salt,” or “pepper,” or “mustard.” In some sense your brain predicts all these possible outcomes at once. However, if I said, “Please pass me the sidewalk,” you would know something is wrong.

Returning to music, we can see probabilistic prediction here as well. If you are listening to a song you have never heard before, you can still have fairly strong expectations. In Western music I expect a regular beat, I expect a repeated rhythm, I expect phrases to last the same number of measures, and I expect songs to end on the tonic pitch. You may not know what these terms mean, but—assuming you have listened to similar music—your brain automatically predicts beats, repeated rhythms, completion of phrases, and ends of songs. If a new song violates these principles, you know immediately that something is wrong. Think about this for a second. You hear a song that you have never heard before, your brain experiences a pattern it has never experienced before, and yet you make predictions and can tell if something is wrong. The basis of these mostly unconscious predictions is the set of memories that are stored in your cortex. Your brain can’t say exactly what will happen next, but it nevertheless predicts which note patterns are likely to happen and which aren’t.

We have all had the experience of suddenly noticing that a source of constant background noise, such as a distant jackhammer or droning Muzak, has just ceased—yet we hadn’t noticed the sound while it was ongoing. Your auditory areas were predicting its continuation, moment after moment, and as long as the noise didn’t change you paid it no heed. By ceasing, it violated your prediction and attracted your attention. Here’s a historical example. Right after New York City stopped running elevated trains, people called the police in the middle of the night claiming that something woke them up. They tended to call around the time the trains used to run past their apartments.

We like to say that seeing is believing. Yet we see what we expect to see as often as we see what we really see. One of the most fascinating examples of this has to do with what researchers call filling in. It may have been brought to your attention before that you have a small blind spot in each eye, where your optic nerve exits each retina through a hole called the optic disk. You have no photoreceptors in this area, so you are permanently blind in the corresponding spot in your visual field. There are two reasons why you don’t usually notice this, one mundane, the other instructive. The mundane reason is that your two blind spots don’t overlap, so one eye compensates for the other.

But interestingly, you still don’t notice your blind spot when only one eye is open. Your visual system “fills in” the missing information. When you close one eye and look at a richly woven Turkish carpet or the wavy contours of wood grain in a cherry tabletop, you don’t see a hole. Entire nodes in the carpet, whole dark knots in the wood grain are constantly winking out of your retina’s view as your blind spot happens to cover them, but your experience is of a seamless stretch of textures and colors. Your visual cortex is drawing on memories of similar patterns and is making a continuous stream of predictions that fill in for any missing input.

Filling in occurs in all parts of the visual image, not just your blind spot. For example, I show you a picture of a shore with a driftwood log lying on some rocks. The boundary between the rocks and the log is clear and obvious. However, if we magnify the image, you will see that the rocks and the log are similar in texture and color where they meet. In the enlarged view, the edge of the log isn’t distinguishable from the rocks at all. If we look at the entire scene, the edge of the log is clear, but in reality we inferred the edge from the rest of the image. When we look at the world, we perceive clean lines and boundaries separating objects, but the raw data entering our eyes are often noisy and ambiguous. Our cortex fills in the missing or messy sections with what it thinks should be there. We perceive an unambiguous image.

Prediction in vision is also a function of the way your eyes move. In chapter 3, I mentioned saccades. About three times every second, your eyes fixate on one point, then suddenly jump to another point. Generally you are not aware of these movements, and you don’t normally consciously control them. And each time your eyes fixate on a new point, the pattern entering your brain from the eyes changes completely from the last fixation. Thus, three times a second your brain sees something completely different. Saccades are not entirely random. When you look at a face your eyes typically fixate first on one eye, then on the other, going back and forth and occasionally fixating on the nose, mouth, ears, and other features. You perceive just “face,” but the eyes see eye, eye, nose, mouth, eye, and so on. I realize it doesn’t feel this way to you. What you’re aware of is a continuous view of the world, but the raw data entering your head are as jerky as a badly wielded Camcorder.

Now imagine that you met someone with an extra nose where an eye should be. Your eyes fixate first on the one eye and then saccade to the second eye, but instead of seeing an eye you see a nose. You would definitely know that something was wrong. For this to happen, your brain has to have an expectation or prediction of what it is about to see. When you predict eye but see nose, the prediction is violated. So several times a second, concurrent with every saccade, your brain makes a prediction about what it will see next. When that prediction is wrong, your attention is immediately aroused. This is why we have difficulty not looking at people with deformities. If you saw a person with two noses, wouldn’t you have trouble not staring? Of course, if you lived with that person, then after a period of time you would get used to two noses and not notice it as unusual anymore.

Think about yourself right now. What predictions are you making? As you turn the pages of this book, you have expectations that the pages bend a certain amount and move in predictable ways that are different from the way the cover moves. If you are sitting, you are predicting that the feelings of pressure on your body will persist; but if the seat turned wet, began drifting backward, or underwent any other unexpected change, you would stop paying attention to the book and try to figure out what is happening. If you spend some time observing yourself, you can begin to understand that your perception of the world, your understanding of the world, is intimately tied to prediction. Your brain has made a model of the world and is constantly checking that model against reality. You know where you are and what you are doing by the validity of this model.

Prediction is not limited to patterns of low-level sensory information like seeing and hearing. Up to now I’ve limited the discussion to such examples because they are the easiest way to introduce this framework for understanding intelligence. However, according to Mountcastle’s principle, what is true of low-level sensory areas must be true for all cortical areas. The human brain is more intelligent than that of other animals because it can make predictions about more abstract kinds of patterns and longer temporal pattern sequences. To predict what my wife will say when she sees me, I must know what she has said in the past, that today is Friday, that the recycling bin has to be put on the curb on Friday nights, that I didn’t do it on time last week, and that her face has a certain look. When she opens her mouth, I have a pretty strong prediction of what she will say. In this case, I don’t know what the exact words will be, but I do know she will be reminding me to take out the recycling. The important point is that higher intelligence is not a different kind of process from perceptual intelligence. It rests fundamentally on the same neocortical memory and prediction algorithm.

Notice that our intelligence tests are in essence prediction tests. From kindergarten through college, I.Q. tests are based on making predictions. Given a sequence of numbers, what should the next number be? Given three different views of a complex object, which of the following is also a view of the object? Word A is to word B as word C is to what word?

Science is itself an exercise in prediction. We advance our knowledge of the world through a process of hypothesis and testing. This book is in essence a prediction about what intelligence is and how brains work. Even product design is fundamentally a predictive process. Whether designing clothes or mobile phones, designers and engineers try to predict what competitors will do, what consumers will want, how much a new design will cost, and what fashions will be in demand.

Intelligence is measured by the capacity to remember and predict patterns in the world, including language, mathematics, physical properties of objects, and social situations. Your brain receives patterns from the outside world, stores them as memories, and makes predictions by combining what it has seen before and what is happening now.

At this point you might be thinking: “I accept that my brain makes predictions and I can be intelligent just lying in the dark. As you point out, I don’t need to act in order to understand or be intelligent. But aren’t situations like that the exception? Are you really arguing that intelligent understanding and behavior are completely separate? In the end, isn’t behavior, not prediction, what makes us intelligent? After all, behavior is the ultimate determiner of survival.”

This is a fair question and of course, in the end, behavior is what matters most to the survival of an animal. Prediction and behavior are not completely separate, but their relationship is subtle. First, the neocortex appeared on the evolutionary scene after animals already evolved sophisticated behaviors. Therefore, the survival value of the cortex must first be understood in terms of the incremental improvements it could bestow upon the animals’ existing behaviors. Behavior came first, then intelligence. Second, most of what we sense is heavily dependent on what we do and how we move in the world. Therefore prediction and behavior are closely related. Let’s look at these issues.

Mammals evolved a large neocortex because it gave them some survival advantage, and such an advantage must ultimately be rooted in behavior. But in the beginning, the cortex served to make more efficient use of existing behaviors, not to create entirely new behaviors. To make the case clear, we need to take a look at how our brains evolved.

Simple nervous systems emerged not long after multicellular creatures started squiggling all over the Earth, hundreds of millions of years ago, but the story of real intelligence begins more recently with our reptilian forebears. The reptiles were successful in their conquest of the land. They spread over every continent and diversified into numerous species. They had keen senses and well-developed brains that endowed them with complex behavior. Their direct descendants, today’s surviving reptiles, still have them. An alligator, for example, has sophisticated senses just like you and me. It has well-developed eyes, ears, nose, mouth, and skin. It carries out complex behaviors including the ability to swim, run, hide, hunt, ambush, sun, nest, and mate.

What is the difference between a human brain and a reptile brain? A lot and a little. I say a little because, to a rough approximation, everything in a reptile’s brain exists in a human brain. I say a lot because a human brain has something really important that a reptile does not have: a large cortex. You sometimes hear people refer to the “old” brain or the “primitive” brain. Every human has these more ancient structures in the brain, just like a reptile. They regulate blood pressure, hunger, sex, emotions, and many aspects of movement. When you stand, balance, and walk, for example, you are relying heavily on the old brain. If you hear a frightening sound, panic, and start to run, that is mostly your old brain. You don’t need more than a reptile brain to do a lot of interesting and useful things. So what does the neocortex do if it isn’t strictly required to see, hear, and move?

Mammals are more intelligent than reptiles because of their neocortex. (The word itself is derived from the Latin words for “new bark” or “new rind,” because the cortex literally covers the old brain.) The neocortex first appeared tens of millions of years ago and only mammals have one. What makes humans smarter than other mammals is primarily the large area of our neocortex—which expanded dramatically only a couple of million years ago. Remember, the cortex is built using a common repeated element. The human cortical sheet is the same thickness and has very nearly the same structure as the cortex in our mammal relatives. When evolution makes something big very quickly, as it did with human cortex, it does so by copying an existing structure. We got smart by adding many more elements of a common cortical algorithm. There is a common misconception that the human brain is the pinnacle of billions of years of evolution. This may be true if we think of the entire nervous system. However, the human neocortex itself is a relatively new structure and hasn’t been around long enough to undergo much long-term evolutionary refinement.

Here then is the core of my argument on how to understand the neocortex, and why memory and prediction are the keys to unlocking the mystery of intelligence. We start with the reptilian brain with no cortex. Evolution discovers that if it tacks on a memory system (the neocortex) to the sensory path of the primitive brain, the animal gains an ability to predict the future. Imagine the old reptilian brain is still doing its thing, but now sensory patterns are simultaneously fed into the neocortex. The neocortex stores this sensory information in its memory. At a future time when the animal encounters the same or a similar situation, the memory recognizes the input as similar and recalls what happened in the past. The recalled memory is compared with the sensory input stream. It both “fills in” the current input and predicts what will be seen next. By comparing the actual sensory input with recalled memory, the animal not only understands where it is but can see into the future.

Now imagine that the cortex not only remembers what the animal has seen but also remembers the behaviors the old brain performed when it was in a similar situation. We don’t even have to assume the cortex knows the difference between sensations and behavior; to the cortex they are both just patterns. When our animal finds itself in the same or a similar situation, it not only sees into the future but recalls which behaviors led to that future vision. Thus, memory and prediction allow an animal to use its existing (old brain) behaviors more intelligently.

For example, imagine you’re a rat learning to navigate a maze for the first time. Aroused by uncertainty or hunger, you will use the skills inherent to your old brain to explore the new environment—listening, looking, sniffing, and creeping close to the walls. All this sensory information is used by your old brain but is also passed up to your neocortex, where it is stored. At some future time, you find yourself in the same maze. Your neocortex will recognize the current input as one it has seen before and recall the stored patterns representing what happened in the past. In essence, it allows you to see a short way into the future. If you were a talking rat, you might say, “Oh, I recognize this maze, and I remember this corner.” As your neocortex recalls what happened in the past, you will envision finding the cheese you saw last time you were in the maze, and how you got to it. “If I turn right here, I know what will happen next. There’s a piece of cheese down at the end of this hallway. I see it in my imagination.” When you scurry through the maze, you rely on older, primitive structures to carry out movements like lifting your feet and sweeping your whiskers. With your (relatively) big neocortex, you can remember the places you have been, recognize them again in the future, and make predictions about what will happen next. A lizard without a neocortex has a much poorer ability to remember the past and may have to search a maze anew every time. You (the rat) understand the world and the immediate future because of your cortical memory. You see vivid images of the rewards and dangers that lie ahead of each decision, and so you move more effectively through your world. You can literally see the future.

But notice you are not performing any particularly complex or fundamentally new behaviors. You are not building yourself a hang glider and flying to the cheese at the end of the hallway. Your neocortex is forming predictions about sensory patterns that allow you to see into the future, but your palette of available behaviors is pretty much unaffected. Your ability to scurry, clamber, and explore is still a lot like that of a lizard.

As the cortex got larger over evolutionary time, it was able to remember more and more about the world. It could form more memories, and make more predictions. The complexity of those memories and predictions also increased. But something else remarkable happened that led to the uniquely human abilities for intelligent behavior.

Human behavior transcends the old basic repertoire of moving around with ratlike skills. We have taken neocortical evolution to a new level. Only humans create written and spoken language. Only humans cook their food, sew clothes, fly planes, and build skyscrapers. Our motor and planning abilities vastly exceed those of our closest animal relatives. How can the cortex, which was designed to make sensory predictions, generate the incredibly sophisticated behavior unique to humans? And how could this superior behavior evolve so suddenly? There are two answers to this question. One is that the neocortical algorithm is so powerful and flexible that with a little bit of rewiring, unique to humans, it can create new, sophisticated behaviors. The other answer is that behavior and prediction are two sides of the same thing. Although the cortex can envision the future, it can make accurate sensory predictions only if it knows what behaviors are being performed.

In the simple example of the rat looking for the cheese, the rat remembers the maze and uses this memory to predict that it will see the cheese around the corner. But the rat could turn left or turn right; only by simultaneously remembering the cheese and the correct behavior, “turn right at the fork,” can the rat make the prediction of the cheese come true. Although this is a trivial example, it gets to the essence of how sensory prediction and behavior are intimately related. All behavior changes what we see, hear, and feel. Most of what we sense at any moment is highly dependent on our own actions. Move your arm in front of your face. To predict seeing your arm, your cortex has to know that it has commanded the arm to move. If the cortex saw your arm moving without the corresponding motor command, you would be surprised. The simplest way to interpret this would be to assume your brain first moves the arm and then predicts what it will see. I believe this is wrong. Instead I believe the cortex predicts seeing the arm, and this prediction is what causes the motor commands to make the prediction come true. You think first, which causes you to act to make your thoughts come true.

Now we want to look at the changes that led to humans having a greatly expanded behavioral repertoire. Are there physical differences between a monkey’s cortex and a human’s cortex that can explain why only humans have language and other complex behaviors? The human brain is about three times larger than the chimpanzee’s. But there is more to it than “bigger is better.” A key to understanding the leap in human behavior is found in the wiring between regions of cortex and parts of the old brain. Put most simply, our brains are connected up differently.

Let’s take a closer look. Everyone is familiar with the brain’s left and right hemispheres. But there is another division that is less well known, and it is where we need to look for human differences. All brains, especially large ones, divide the cortex into a front half and a back half. Scientists use the words anterior for the front and posterior for the back. Separating the front and the back is a large fissure called the central sulcus. The back part of the cortex contains the sections where the eyes, ears, and touch inputs arrive. It is where sensory perception largely occurs. The front part contains regions of cortex that are involved in highlevel planning and thought. It also contains the motor cortex, the section of brain most responsible for moving muscles and therefore creating behavior.

As the primate neocortex became larger over time, the anterior half got disproportionately larger, especially so in humans. Compared with other primates and early hominids, we have enormous foreheads designed to contain our very large anterior cortex. But this enlargement alone is not enough to explain the improvement in our motor ability as compared with that of other creatures. Our ability to make exceptionally complex movements stems from the fact that our motor cortex makes many more connections with the muscles in our bodies. In other mammals, the front cortex plays a less direct role in motor behavior. Most animals rely largely on the older parts of the brain for generating their behavior. In contrast, the human cortex usurped most of the motor control from the rest of the brain. If you damage the motor cortex of a rat, the rat may not have noticeable deficits. If you damage the motor cortex of a human, he or she becomes paralyzed.

People often ask me about dolphins. Don’t they have huge brains? The answer is yes; a dolphin has a large neocortex. Dolphin cortex has a simpler structure (three layers versus our six) than a human neocortex, but by any other measure it is large. It is likely a dolphin can remember and understand lots of things. It can recognize other individual dolphins. It probably has an excellent memory of its own life, in an autobiographical sense. It probably knows every nook and cranny of the ocean it’s ever been to. But although they exhibit some sophisticated behavior, dolphins don’t come close to our own. So we can surmise their cortex has a less-dominant influence on their behavior. The point is that the cortex evolved primarily to provide a memory of the world. An animal with a large cortex could perceive the world much as you and I do. But humans are unique in the dominant, advanced role the cortex plays in our behavior. It is why we have complex language and intricate tools whereas other animals don’t. It is why we can write novels, surf the Internet, send probes to Mars, and build cruise ships.

Now we can see the entire picture. Nature first created animals such as reptiles with sophisticated senses and sophisticated but relatively rigid behaviors. It then discovered that by adding a memory system and feeding the sensory stream into it, the animal could remember past experiences. When the animal found itself in the same or a similar situation, the memory would be recalled, leading to a prediction of what was likely to happen next. Thus, intelligence and understanding started as a memory system that fed predictions into the sensory stream. These predictions are the essence of understanding. To know something means that you can make predictions about it.

The cortex evolved in two directions. First it got larger and more sophisticated in the types of memories it could store; it was able to remember more things and make predictions based on more complex relationships. Second, it started interacting with the motor system of the old brain. To predict what you will hear, see, and feel next, it needed to know what actions were being taken. With humans the cortex has taken over most of our motor behavior. Instead of just making predictions based on the behavior of the old brain, the human neocortex directs behavior to satisfy its predictions.

The human cortex is particularly large and therefore has a massive memory capacity. It is constantly predicting what you will see, hear, and feel, mostly in ways you are unconscious of. These predictions are our thoughts, and, when combined with sensory input, they are our perceptions. I call this view of the brain the memory-prediction framework of intelligence.

If Searle’s Chinese Room contained a similar memory system that could make predictions about what Chinese characters would appear next and what would happen next in the story, we could say with confidence that the room understood Chinese and understood the story. We can now see where Alan Turing went wrong. Prediction, not behavior, is the proof of intelligence.

We are now ready to delve into the details of this new idea of the memory-prediction framework of the brain. To make predictions of future events, your neocortex has to store sequences of patterns. To recall the appropriate memories, it has to retrieve patterns by their similarity to past patterns (auto-associative recall). And, finally, memories have to be stored in an invariant form so that the knowledge of past events can be applied to new situations that are similar but not identical to the past. How the physical cortex accomplishes these tasks, plus a fuller exploration of its hierarchy, is the subject of the next chapter.