{"componentChunkName":"component---src-templates-book-page-js","path":"/intelligence/7/","result":{"data":{"mdx":{"id":"54646f9d-8966-50f2-a205-bd04def20166","body":"function _extends() { _extends = Object.assign || function (target) { for (var i = 1; i < arguments.length; i++) { var source = arguments[i]; for (var key in source) { if (Object.prototype.hasOwnProperty.call(source, key)) { target[key] = source[key]; } } } return target; }; return _extends.apply(this, arguments); }\n\nfunction _objectWithoutProperties(source, excluded) { if (source == null) return {}; var target = _objectWithoutPropertiesLoose(source, excluded); var key, i; if (Object.getOwnPropertySymbols) { var sourceSymbolKeys = Object.getOwnPropertySymbols(source); for (i = 0; i < sourceSymbolKeys.length; i++) { key = sourceSymbolKeys[i]; if (excluded.indexOf(key) >= 0) continue; if (!Object.prototype.propertyIsEnumerable.call(source, key)) continue; target[key] = source[key]; } } return target; }\n\nfunction _objectWithoutPropertiesLoose(source, excluded) { if (source == null) return {}; var target = {}; var sourceKeys = Object.keys(source); var key, i; for (i = 0; i < sourceKeys.length; i++) { key = sourceKeys[i]; if (excluded.indexOf(key) >= 0) continue; target[key] = source[key]; } return target; }\n\n/* @jsxRuntime classic */\n\n/* @jsx mdx */\nvar _frontmatter = {\n  \"author\": \"Jeff Hawkins\",\n  \"bookTitle\": \"On Intelligence\",\n  \"isBook\": false,\n  \"numSections\": 16,\n  \"tags\": [\"a\"],\n  \"templateKey\": \"book-page\",\n  \"title\": \"4. MEMORY\"\n};\nvar layoutProps = {\n  _frontmatter: _frontmatter\n};\nvar MDXLayout = \"wrapper\";\nreturn function MDXContent(_ref) {\n  var components = _ref.components,\n      props = _objectWithoutProperties(_ref, [\"components\"]);\n\n  return mdx(MDXLayout, _extends({}, layoutProps, props, {\n    components: components,\n    mdxType: \"MDXLayout\"\n  }), mdx(ContentRef, {\n    id: 0,\n    mdxType: \"ContentRef\"\n  }, mdx(\"strong\", null, \"As\"), \" you read this book, walk down a crowded street, hear a symphony, or comfort a crying child, your brain is being flooded with the spatial and temporal patterns from all of your senses. The world is an ocean of constantly changing patterns that come lapping and crashing into your brain. How do you manage to make sense of the onslaught? Patterns stream in, pass through various parts of the old brain, and eventually arrive at the neocortex. But what happens to them when they enter the cortex?\"), mdx(ContentRef, {\n    id: 1,\n    mdxType: \"ContentRef\"\n  }, \"From the dawn of the industrial revolution, people have viewed the brain as some sort of machine. They knew there weren\\u2019t gears and cogs in the head, but it was the best metaphor they had. Somehow information entered the brain and the brain-machine determined how the body should react. During the computer age, the brain has been viewed as a particular type of machine, the programmable computer. And as we saw in \", mdx(\"a\", {\n    className: \"nounder\",\n    href: \"ch01.html#ch01\"\n  }, \"chapter 1\"), \", AI researchers have stuck with this view, arguing that their lack of progress is only due to how small and slow computers remain compared \", mdx(\"a\", {\n    id: \"page_66\"\n  }), \"to the human brain. Today\\u2019s computers may be equivalent only to a cockroach brain, they say, but when we make bigger and faster computers they will be as intelligent as humans.\"), mdx(ContentRef, {\n    id: 2,\n    mdxType: \"ContentRef\"\n  }, \"There is a largely ignored problem with this brain-as-computer analogy. Neurons are quite slow compared to the transistors in a computer. A neuron collects inputs from its synapses, and combines these inputs together to decide when to output a spike to other neurons. A typical neuron can do this and reset itself in about five milliseconds (5 ms), or around two hundred times per second. This may seem fast, but a modern silicon-based computer can do one billion operations in a second. This means a basic computer operation is five million times faster than the basic operation in your brain! That is a very, very big difference. So how is it possible that a brain could be faster and more powerful than our fastest digital computers? \\u201CNo problem,\\u201D say the brain-as-computer people. \\u201CThe brain is a parallel computer. It has billions of cells all computing at the same time. This parallelism vastly multiplies the processing power of the biological brain.\\u201D\"), mdx(ContentRef, {\n    id: 3,\n    mdxType: \"ContentRef\"\n  }, \"I always felt this argument was a fallacy, and a simple thought experiment shows why. It is called the \\u201Cone hundred-step rule.\\u201D A human can perform significant tasks in much less time than a second. For example, I could show you a photograph and ask you to determine if there is cat in the image. Your job would be to push a button if there is a cat, but not if you see a bear or a warthog or a turnip. This task is difficult or impossible for a computer to perform today, yet a human can do it reliably in half a second or less. But neurons are slow, so in that half a second, the information entering your brain can only traverse a chain one hundred neurons long. That is, the brain \\u201Ccomputes\\u201D solutions to problems like this in one hundred steps or fewer, regardless of how many total neurons might be involved. From the time light enters your eye to the time you press the button, a chain no longer than one hundred neurons could be involved. A \", mdx(\"a\", {\n    id: \"page_67\"\n  }), \"digital computer attempting to solve the same problem would take billions of steps. One hundred computer instructions are barely enough to move a single character on the computer\\u2019s display, let alone do something interesting.\"), mdx(ContentRef, {\n    id: 4,\n    mdxType: \"ContentRef\"\n  }, \"But if I have many millions of neurons working together, isn\\u2019t that like a parallel computer? Not really. Brains operate in parallel and parallel computers operate in parallel, but that\\u2019s the only thing they have in common. Parallel computers combine many fast computers to work on large problems such as computing tomorrow\\u2019s weather. To predict the weather you have to compute the physical conditions at many points on the planet. Each computer can work on a different location at the same time. But even though there may be hundreds or even thousands of computers working in parallel, the individual computers still need to perform billions or trillions of steps to accomplish their task. The largest conceivable parallel computer can\\u2019t do anything useful in one hundred steps, no matter how large or how fast.\"), mdx(ContentRef, {\n    id: 5,\n    mdxType: \"ContentRef\"\n  }, \"Here is an analogy. Suppose I ask you to carry one hundred stone blocks across a desert. You can carry one stone at a time and it takes a million steps to cross the desert. You figure this will take a long time to complete by yourself, so you recruit a hundred workers to do it in parallel. The task now goes a hundred times faster, but it still requires a minimum of a million steps to cross the desert. Hiring more workers\\u2014even a thousand workers\\u2014wouldn\\u2019t provide any additional gain. No matter how many workers you hire, the problem cannot be solved in less time than it takes to walk a million steps. The same is true for parallel computers. After a point, adding more processors doesn\\u2019t make a difference. A computer, no matter how many processors it might have and no matter how fast it runs, cannot \\u201Ccompute\\u201D the answer to difficult problems in one hundred steps.\"), mdx(ContentRef, {\n    id: 6,\n    mdxType: \"ContentRef\"\n  }, \"So how can a brain perform difficult tasks in one hundred steps that the largest parallel computer imaginable can\\u2019t solve in \", mdx(\"a\", {\n    id: \"page_68\"\n  }), \"a million or a billion steps? The answer is the brain doesn\\u2019t \\u201Ccompute\\u201D the answers to problems; it retrieves the answers from memory. In essence, the answers were stored in memory a long time ago. It only takes a few steps to retrieve something from memory. Slow neurons are not only fast enough to do this, but they constitute the memory themselves. The entire cortex is a memory system. It isn\\u2019t a computer at all.\"), mdx(ContentRef, {\n    id: 7,\n    mdxType: \"ContentRef\"\n  }, \"Let me show, through an example, the difference between \", mdx(\"em\", null, \"computing\"), \" a solution to a problem and \", mdx(\"em\", null, \"using memory\"), \" to solve the same problem. Consider the task of catching a ball. Someone throws a ball to you, you see it traveling toward you, and in less than a second you snatch it out of the air. This doesn\\u2019t seem too difficult\\u2014until you try to program a robot arm to do the same. As many a graduate student has found out the hard way, it seems nearly impossible. When engineers or computer scientists tackle this problem, they first try to calculate the flight of the ball to determine where it will be when it reaches the arm. This calculation requires solving a set of equations of the type you learn in high school physics. Next, all the joints of a robotic arm have to be adjusted in concert to move the hand into the proper position. This involves solving another set of mathematical equations more difficult than the first. Finally, this whole operation has to be repeated multiple times, for as the ball approaches, the robot gets better information about the ball\\u2019s location and trajectory. If the robot waits to start moving until it knows exactly where the ball will arrive it will be too late to catch it. It has to start moving to catch the ball when it has only a poor sense of its location and it continually adjusts as the ball gets closer. A computer requires millions of steps to solve the numerous mathematical equations to catch the ball. And although a computer might be programmed to successfully solve this problem, the one hundred-step rule tells us that a brain solves it in a different way. It uses memory.\"), mdx(ContentRef, {\n    id: 8,\n    mdxType: \"ContentRef\"\n  }, mdx(\"a\", {\n    id: \"page_69\"\n  }), \"How do you catch the ball using memory? Your brain has a stored memory of the muscle commands required to catch a ball (along with many other learned behaviors). When a ball is thrown, three things happen. First, the appropriate memory is automatically recalled by the sight of the ball. Second, the memory actually recalls a temporal sequence of muscle commands. And third, the retrieved memory is adjusted as it is recalled to accommodate the particulars of the moment, such as the ball\\u2019s actual path and the position of your body. The memory of how to catch a ball was not programmed into your brain; it was learned over years of repetitive practice, and it is stored, not calculated, in your neurons.\"), mdx(ContentRef, {\n    id: 9,\n    mdxType: \"ContentRef\"\n  }, \"You might be thinking, \\u201CWait a minute. Each catch is slightly different. You just said the recalled memory gets continually adjusted to accommodate the variations of where the ball is on any particular throw \\u2026 Doesn\\u2019t that require solving the same equations we were trying to avoid?\\u201D It may seem so, but nature solved the problem of variation in a different and very clever way. As we\\u2019ll see later in this chapter, the cortex creates what are called \", mdx(\"em\", null, \"invariant representations,\"), \" which handle variations in the world automatically. A helpful analogy might be to imagine what happens when you sit down on a water bed: the pillows and any other people on the bed are all spontaneously pushed into a new configuration. The bed doesn\\u2019t compute how high each object should be elevated; the physical properties of the water and the mattress\\u2019s plastic skin take care of the adjustment automatically. As we\\u2019ll see in the next chapter, the design of the six-layered cortex does something similar, loosely speaking, with the information that flows through it.\"), mdx(ContentRef, {\n    id: 10,\n    mdxType: \"ContentRef\"\n  }, \"So the neocortex is not like a computer, parallel or otherwise. Instead of computing answers to problems the neocortex uses stored memories to solve problems and produce behavior.\"), mdx(ContentRef, {\n    id: 11,\n    mdxType: \"ContentRef\"\n  }, mdx(\"a\", {\n    id: \"page_70\"\n  }), \"Computers have memory too, in the form of hard drives and memory chips; however, there are four attributes of neocortical memory that are fundamentally different from computer memory:\"), mdx(ContentRef, {\n    id: 12,\n    mdxType: \"ContentRef\"\n  }, \"\\u2022 \", mdx(\"strong\", null, \"The neocortex stores sequences of patterns.\")), mdx(ContentRef, {\n    id: 13,\n    mdxType: \"ContentRef\"\n  }, \"\\u2022 \", mdx(\"strong\", null, \"The neocortex recalls patterns auto-associatively.\")), mdx(ContentRef, {\n    id: 14,\n    mdxType: \"ContentRef\"\n  }, \"\\u2022 \", mdx(\"strong\", null, \"The neocortex stores patterns in an invariant form.\")), mdx(ContentRef, {\n    id: 15,\n    mdxType: \"ContentRef\"\n  }, \"\\u2022 \", mdx(\"strong\", null, \"The neocortex stores patterns in a hierarchy.\")), mdx(ContentRef, {\n    id: 16,\n    mdxType: \"ContentRef\"\n  }, \"We will discuss the first three differences in this chapter. I introduced the concept of hierarchy in the neocortex in \", mdx(\"a\", {\n    className: \"nounder\",\n    href: \"ch03.html#ch03\"\n  }, \"chapter 3\"), \". In \", mdx(\"a\", {\n    className: \"nounder\",\n    href: \"ch06.html#ch06\"\n  }, \"chapter 6\"), \", I will describe its significance and how it works.\"), mdx(ContentRef, {\n    id: 17,\n    mdxType: \"ContentRef\"\n  }, \"The next time you tell a story, step back and consider how you can only relate one aspect of the tale at a time. You cannot tell me everything that happened all at once, no matter how quickly you talk or I listen. You need to finish one part of the story before you can move on to the next. This isn\\u2019t only because spoken language is serial; written, oral, and visual storytelling all convey a narrative in a serial fashion. It is because the story is stored in your head in a sequential fashion and can only be recalled in the same sequence. You can\\u2019t remember the entire story at once. In fact, it\\u2019s almost impossible to think of anything complex that isn\\u2019t a series of events or thoughts.\"), mdx(ContentRef, {\n    id: 18,\n    mdxType: \"ContentRef\"\n  }, \"You may have noticed, too, that in telling a story some people can\\u2019t get to the crux of it right away. They seem to ramble on with irrelevant details and tangents. This can be irritating. You want to scream, \\u201CGet to the point!\\u201D But they are chronicling the story as it happened to them, through time, and cannot tell it any other way.\"), mdx(ContentRef, {\n    id: 19,\n    mdxType: \"ContentRef\"\n  }, \"Another example: I\\u2019d like you to imagine your home right now. Close your eyes and visualize it. In your imagination, go to the front door. Imagine what it looks like. Open your front door. Move inside. Now look to your left. What do you see? Look to \", mdx(\"a\", {\n    id: \"page_71\"\n  }), \"the right. What is there? Go to your bathroom. What\\u2019s on the right? What\\u2019s on the left? What\\u2019s in the top right drawer? What items do you keep in your shower? You know all these things plus thousands more and can recall them in great detail. These memories are stored in your cortex. You might say these things are all part of the memory of your home. But you can\\u2019t think of them all at once. They are obviously related memories but there is no way you can bring to mind all of this detail at once. You have a thorough memory of your home; but to recall it you have to go through it in sequential segments, in much the same way as you experience it.\"), mdx(ContentRef, {\n    id: 20,\n    mdxType: \"ContentRef\"\n  }, \"All memories are like this. You have to walk through the temporal sequence of how you do things. One pattern (approach the door) evokes the next pattern (go through the door), which evokes the next pattern (either go down the hall or ascend the stairs), and so on. Each is a sequence you\\u2019ve followed before. Of course, with a conscious effort I can change the order of how I describe my home to you. I can jump from basement to the second floor if I decide to focus on items in a nonsequential way. Yet once I start to describe any room or item I\\u2019ve chosen, I\\u2019m back to following a sequence. Truly random thoughts don\\u2019t exist. Memory recall almost always follows a pathway of association.\"), mdx(ContentRef, {\n    id: 21,\n    mdxType: \"ContentRef\"\n  }, \"You know the alphabet. Try saying it backward. You can\\u2019t because you don\\u2019t usually experience it backward. If you want to know what it\\u2019s like to be a child learning the alphabet, try saying it in reverse. That\\u2019s exactly what they\\u2019re confronted with. It\\u2019s really hard. Your memory of the alphabet is a sequence of patterns. It isn\\u2019t something stored or recalled in an instant or in an arbitrary order. The same thing goes for the days of the week, the months of the year, your phone number, and countless other things.\"), mdx(ContentRef, {\n    id: 22,\n    mdxType: \"ContentRef\"\n  }, \"Your memory for songs is a great example of temporal sequences in memory. Think of a tune you know. I like to use \", mdx(\"a\", {\n    id: \"page_72\"\n  }), \"\\u201CSomewhere over the Rainbow,\\u201D but any melody will suffice. You cannot imagine the entire song at once, only in sequence. You can start at the beginning or maybe with the chorus, and then you play through it, filling in the notes one after another. You can\\u2019t recall the song backward, just as you can\\u2019t recall it all at once. You were first exposed to \\u201CSomewhere over the Rainbow\\u201D as it played through time, and you can only recall it in the same way you learned it.\"), mdx(ContentRef, {\n    id: 23,\n    mdxType: \"ContentRef\"\n  }, \"This applies to very low level sensory memories too. Consider your tactile memory for textures. Your cortex has memories of what it feels like to hold a fistful of gravel, slide your fingers over velvet, and press down on a piano key. These memories are based on sequences every bit as much as the alphabet and songs are; it\\u2019s just that the sequences are shorter, spanning mere fractions of a second rather than many seconds or minutes. If I buried your hand in a bucket of gravel while you slept, when you woke up you wouldn\\u2019t know what you were touching until you moved your fingers. Your memory for the tactile texture of gravel is based on pattern sequences across the pressure-and vibration-sensing neurons in your skin. These sequences are different from those you\\u2019d receive if your hand was buried in sand or Styrofoam pellets or dry leaves. As soon as you flexed your hand, the scraping and rolling of the pebbles would create the telltale pattern sequences of gravel and trigger the appropriate memory in your somatosensory cortex.\"), mdx(ContentRef, {\n    id: 24,\n    mdxType: \"ContentRef\"\n  }, \"The next time you get out of the shower, pay attention to how you dry yourself off with a towel. I discovered that I dry myself off with nearly the exact same sequence of rubs, pats, and body positions each time. And via a pleasant experiment I discovered that my wife also follows a semirigid pattern when she steps out of the shower. You probably do too. If you follow a sequence, try changing it. You can will yourself to do it, but you need to stay focused. If your attention wanders, you\\u2019ll fall back into your accustomed pattern.\"), mdx(ContentRef, {\n    id: 25,\n    mdxType: \"ContentRef\"\n  }, mdx(\"a\", {\n    id: \"page_73\"\n  }), \"All memories are stored in the synaptic connections between neurons. Given the very large number of things we have stored in our cortex, and that at any moment in time we can recall only a tiny fraction of these stored memories, it stands to reason that only a limited number of synapses and neurons in your brain are playing an active role in memory recall at any one time. As you start to recall what is in your home, one set of neurons becomes active, which then leads to another set of neurons being active, and so on. An adult human neocortex has an incredibly large memory capacity. But, even though we have stored so many things, we can only remember a few at any time and can only do so in a sequence of associations.\"), mdx(ContentRef, {\n    id: 26,\n    mdxType: \"ContentRef\"\n  }, \"Here is a fun exercise. Try to recall details from your past, details of where you lived, places you visited, and people you knew. I find I can always uncover memories of things I haven\\u2019t thought of in many years. There are thousands of detailed memories stored in the synapses of our brains that are rarely used. At any point in time we recall only a tiny fraction of what we know. Most of the information is sitting there idly waiting for the appropriate cues to invoke it.\"), mdx(ContentRef, {\n    id: 27,\n    mdxType: \"ContentRef\"\n  }, \"Computer memory does not normally store sequences of patterns. It can be made to do so using various software tricks (such as when you store a song on your computer), but computer memory does not do this automatically. In contrast, the cortex does store sequences automatically. Doing so is an inherent aspect of the neocortical memory system.\"), mdx(ContentRef, {\n    id: 28,\n    mdxType: \"ContentRef\"\n  }, \"Now let\\u2019s consider the second key feature of our memory, its auto-associative nature. As we saw in \", mdx(\"a\", {\n    className: \"nounder\",\n    href: \"ch02.html#ch02\"\n  }, \"chapter 2\"), \", the term simply means that patterns are associated with themselves. An auto-associative memory system is one that can recall complete patterns when given only partial or distorted inputs. This can work for both spatial and temporal patterns. If you see your child\\u2019s \", mdx(\"a\", {\n    id: \"page_74\"\n  }), \"shoes sticking out from behind the draperies, you automatically envision his or her entire form. You complete the spatial pattern from a partial version of it. Or imagine you see a person waiting for a bus but can only see part of her because she is standing partially behind a bush. Your brain is not confused. Your eyes only see parts of a body, but your brain fills in the rest, creating a perception of a whole person that\\u2019s so strong you may not even realize you\\u2019re only inferring.\"), mdx(ContentRef, {\n    id: 29,\n    mdxType: \"ContentRef\"\n  }, \"You also complete temporal patterns. If you recall a small detail about something that happened long ago, the entire memory sequence can come flooding back into your mind. Marcel Proust\\u2019s famous series of novels, \", mdx(\"em\", null, \"Remembrance of Things Past,\"), \" opened with the memory of how a madeleine cookie smelled\\u2014and he was off and running for a thousand-plus pages. During conversation we often can\\u2019t hear all the words if we are in a noisy environment. No problem. Our brains fill in what they miss with what they expect to hear. It\\u2019s well established that we don\\u2019t actually hear all the words we perceive. Some people complete others\\u2019 sentences aloud, but in our minds all of us are doing this constantly. And not just the ends of sentences, but the middles and beginnings as well. For the most part we are not aware that we\\u2019re constantly completing patterns, but it\\u2019s a ubiquitous and fundamental feature of how memories are stored in the cortex. At any time, a piece can activate the whole. This is the essence of auto-associative memories.\"), mdx(ContentRef, {\n    id: 30,\n    mdxType: \"ContentRef\"\n  }, \"Your neocortex is a complex biological auto-associative memory. During each waking moment, each functional region is essentially waiting vigilantly for familiar patterns or pattern fragments to come in. You can be in deep thought about something, but the instant your friend appears your thoughts switch to her. This switch isn\\u2019t something you chose to do. The mere appearance of your friend forces your brain to start recalling patterns associated with her. It\\u2019s unavoidable. After an interruption we frequently have to ask, \\u201CWhat was I thinking about?\\u201D A dinner \", mdx(\"a\", {\n    id: \"page_75\"\n  }), \"conversation with friends follows a circuitous route of associations. The talk may start with the food in front of you, but the salad evokes an associated memory of your mother\\u2019s salad at your wedding, which leads to a memory of someone else\\u2019s wedding, which leads to a memory of where they went on their honeymoon, to the political problems in that part of the world, and so on. Thoughts and memories are associatively linked, and again, random thoughts never really occur. Inputs to the brain auto-associatively link to themselves, filling in the present, and auto-associatively link to what normally follows next. We call this chain of memories \", mdx(\"em\", null, \"thought,\"), \" and although its path is not deterministic, we are not fully in control of it either.\"), mdx(ContentRef, {\n    id: 31,\n    mdxType: \"ContentRef\"\n  }, \"Now we can consider the third major attribute of neocortical memory: how it forms what are called invariant representations. I will cover the basic ideas of invariant representations in this chapter and, in \", mdx(\"a\", {\n    className: \"nounder\",\n    href: \"ch06.html#ch06\"\n  }, \"chapter 6\"), \", the details of how the cortex creates them.\"), mdx(ContentRef, {\n    id: 32,\n    mdxType: \"ContentRef\"\n  }, \"A computer\\u2019s memory is designed to store information exactly as it is presented. If you copy a program from a CD to a hard disk, every byte is copied with 100 percent fidelity. A single error or discrepancy between the two copies might cause the program to crash. The memory in our neocortex is different. Our brain does not remember exactly what it sees, hears, or feels. We don\\u2019t remember or recall things with complete fidelity\\u2014not because the cortex and its neurons are sloppy or error-prone but because the brain remembers the important relationships in the world, independent of the details. Let\\u2019s look at several examples to illustrate this point.\"), mdx(ContentRef, {\n    id: 33,\n    mdxType: \"ContentRef\"\n  }, \"As we saw in \", mdx(\"a\", {\n    className: \"nounder\",\n    href: \"ch02.html#ch02\"\n  }, \"chapter 2\"), \", simple auto-associative memory models have been around for decades and, as I described above, the brain recalls memories auto-associatively. But there is a big difference between the auto-associative memories built by neural \", mdx(\"a\", {\n    id: \"page_76\"\n  }), \"network researchers and those in the cortex. Artificial auto-associative memories do not use invariant representations and therefore they fail in some very basic ways. Imagine I have a picture of a face formed by a large collection of black-and-white dots. This picture is a pattern, and if I have an artificial autoassociative memory I can store many pictures of faces in the memory. Our artificial auto-associative memory is robust in that if I give it half a face or just a pair of eyes, it will recognize that part of the image and fill in the missing parts correctly. This exact experiment has been done several times. However, if I move each dot in the picture five pixels to the left, the memory completely fails to recognize the face. To the artificial autoassociative memory, it is a completely novel pattern, because none of the pixels between the previously stored pattern and the new pattern are aligned. You and I, of course, would have no difficulty seeing the shifted pattern as the same face. We probably wouldn\\u2019t even notice the change. Artificial auto-associative memories fail to recognize patterns if they are moved, rotated, rescaled, or transformed in any of a thousand other ways, whereas our brains handle these variations with ease. How can we perceive something as being the same or constant when the input patterns representing it are novel and changing? Let\\u2019s look at another example.\"), mdx(ContentRef, {\n    id: 34,\n    mdxType: \"ContentRef\"\n  }, \"You are probably holding a book in your hands right now. As you move the book, or change the lighting, or reposition yourself in your chair, or fixate your eyes on different parts of the page, the pattern of light falling on your retina changes completely. The visual input you receive is different moment by moment and never repeats. In fact, you could hold this book for a hundred years and not once would the pattern on your retina, and therefore the pattern entering your brain, be exactly the same. Yet not for an instant do you have any doubt that you are holding a book, indeed the same book. Your brain\\u2019s internal pattern representing \\u201Cthis book\\u201D does not change even though \", mdx(\"a\", {\n    id: \"page_77\"\n  }), \"the stimuli informing you it\\u2019s there are in constant flux. Hence we use the term \", mdx(\"em\", null, \"invariant representation\"), \" to refer to the brain\\u2019s internal representation.\"), mdx(ContentRef, {\n    id: 35,\n    mdxType: \"ContentRef\"\n  }, \"For another example, think of a friend\\u2019s face. You recognize her every time you see her. It happens automatically in less than a second. It doesn\\u2019t matter if she is two feet away, three feet away, or across the room. When she is close, her image occupies most of your retina. When she is far away, her image occupies a small portion of your retina. She can be facing you, turned a little to the side, or in profile. She might be smiling, squinting, or yawning. You might see her in bright light, in shade, or under strangely angled disco lights. Her visage can appear in countless positions and variations. For each one, the pattern of light falling on your retina is unique, yet in every case you know instantly that you are looking at her.\"), mdx(ContentRef, {\n    id: 36,\n    mdxType: \"ContentRef\"\n  }, \"Let\\u2019s pop the hood and look at what\\u2019s going on in your brain to perform this amazing feat. We know from experiments that if we monitor the activity of neurons in the visual input area of your cortex, called V1, the pattern of activity is different for each different view of her face. Every time the face moves or your eyes make a new fixation, the pattern of activity in V1 changes, much like the changing pattern on the retina. However, if we monitor the activity of cells in your face recognition area\\u2014a functional region that\\u2019s several steps higher than V1 in the cortical hierarchy\\u2014we find stability. That is, some set of the cells in the face recognition area remain active as long as your friend\\u2019s face is anywhere in your field of vision (or even being conjured in your mind\\u2019s eye), regardless of its size, position, orientation, scale, and expression. This stability of cell firing is an invariant representation.\"), mdx(ContentRef, {\n    id: 37,\n    mdxType: \"ContentRef\"\n  }, \"Introspectively, this task seems so easy as to be hardly worth calling it a problem. It\\u2019s as automatic as breathing. It seems trivial because we aren\\u2019t consciously aware it is happening. And in some sense, it \", mdx(\"em\", null, \"is\"), \" trivial because our brains can solve it so quickly \", mdx(\"a\", {\n    id: \"page_78\"\n  }), \"(remember the one hundred-step rule). However, the problem of understanding how your cortex forms invariant representations remains one of the biggest mysteries in all of science. How difficult, you ask? So much so that no one, not even using the most powerful computers in the world, has been able to solve it. And it isn\\u2019t for a lack of trying.\"), mdx(ContentRef, {\n    id: 38,\n    mdxType: \"ContentRef\"\n  }, \"Speculation on this problem has an ancient pedigree. It traces back to Plato, twenty-three centuries ago. Plato wondered how people are able to think and know about the world. He pointed out that real-world instances of things and ideas are always imperfect and are always different. For example, you have a concept of a perfect circle, yet you have never actually seen one. All drawings of circles are imperfect. Even if drafted with a geometer\\u2019s compass a so-called circle is represented by a dark line, whereas the circumference of a true circle has no thickness at all. How then did you ever acquire the concept of a perfect circle? Or to take a more worldly case, think about your concept of dogs. Every dog you\\u2019ve ever seen is different from every other, and every time you see the same individual dog you see a different view of it. All dogs are different and you can never see any particular dog exactly the same way twice. Yet all of your various experiences with dogs get funneled into a mental concept of \\u201Cdog\\u201D that is stable across all of them. Plato was perplexed. How is it possible that we learn and apply concepts in this world of infinitely various forms and ever-shifting sensations?\"), mdx(ContentRef, {\n    id: 39,\n    mdxType: \"ContentRef\"\n  }, \"Plato\\u2019s solution was his famous Theory of Forms. He concluded that our higher minds must be tethered to some transcendent plane of superreality, where fixed, stable ideas (Forms with a capital F) exist in timeless perfection. Our souls come from this mystical place before birth, he decided, which is where they learned about the Forms in the first place. After we\\u2019re born we retain latent knowledge of them. Learning and understanding happen because real-world forms remind us of the Forms to which they correspond. You are able to know \", mdx(\"a\", {\n    id: \"page_79\"\n  }), \"about circles and dogs because they respectively trigger your soul memories of Circle and Dog.\"), mdx(ContentRef, {\n    id: 40,\n    mdxType: \"ContentRef\"\n  }, \"It\\u2019s all quite loopy from a modern perspective. But if you strip away the high-flown metaphysics, you can see that he was really talking about invariance. His system of explanation was wildly off the mark, but his intuition that this was one of the most important questions we can ask about our own nature was a bull\\u2019s-eye.\"), mdx(ContentRef, {\n    id: 41,\n    mdxType: \"ContentRef\"\n  }, \"Lest you get the impression that invariance is all about vision, let\\u2019s look at some examples in other senses. Consider your tactile sense. When you reach into your car\\u2019s glove compartment to find your sunglasses, your fingers only have to brush against them for you to know you\\u2019ve found them. It doesn\\u2019t matter which part of your hand makes the contact; it can be your thumb, any part of any finger, or your palm. And the contact can be with any part of the glasses, whether it\\u2019s a lens, temple, hinge, or part of the frame. Just a second of moving any part of your hand over any portion of the glasses is sufficient for your brain to identify them. In each case, the stream of spatial and temporal patterns coming from your touch receptors is entirely different\\u2014different areas of your skin, different parts of the object\\u2014yet you snap up your sunglasses without a thought.\"), mdx(ContentRef, {\n    id: 42,\n    mdxType: \"ContentRef\"\n  }, \"Or consider the sensorimotor task of putting the key in your car\\u2019s ignition switch. The position of your seat, body, arm, and hand are slightly different each time. To you it feels like the same simple repetitive action day in, day out, but that\\u2019s because you have an invariant representation of it in your brain. If you tried to make a robot that could enter the car and put in the key, you would quickly see how nearly impossible it is unless you made sure the robot was in the exact same position, and held the key in exactly the same way every time. And even if you could manage to do this, the robot would need to be reprogrammed for \", mdx(\"a\", {\n    id: \"page_80\"\n  }), \"different cars. Robots and computer programs, like artificial auto-associative memories, are terrible at handling variation.\"), mdx(ContentRef, {\n    id: 43,\n    mdxType: \"ContentRef\"\n  }, \"Another interesting example is your signature. Somewhere in your motor cortex, in your frontal lobe, you have an invariant representation of your autograph. Every time you sign your name, you use the same sequence of strokes, angles, and rhythms. This is true whether you sign it minutely with a fine-tipped pen, flamboyantly like John Hancock, in the air with your elbow, or clumsily with a pencil held between your toes. It comes out looking somewhat different each time, of course, especially under some of the awkward conditions I just named. Nevertheless, regardless of scale, writing implement, or combination of body parts, you always run the same abstract \\u201Cmotor program\\u201D to produce it.\"), mdx(ContentRef, {\n    id: 44,\n    mdxType: \"ContentRef\"\n  }, \"From the signature example you can see that invariant representation in motor cortex is, in some ways, the mirror image of invariant representation in sensory cortex. On the sensory side, a wide variety of input patterns can activate a stable cell assembly that represents some abstract pattern (your friend\\u2019s face, your sunglasses). On the motor side, a stable cell assembly representing some abstract motor command (catching a ball, signing your name) is able to express itself using a wide variety of muscle groups and respecting a wide variety of other constraints. This symmetry between perception and action is what we should expect if, as Mountcastle proposed, the cortex runs a single basic algorithm in all areas.\"), mdx(ContentRef, {\n    id: 45,\n    mdxType: \"ContentRef\"\n  }, \"For a final example, let\\u2019s return to sensory cortex and look at music again. (I like using memory of music as an example because it is easy to see all the issues the neocortex must solve.) Invariant representation in music is illustrated by your ability to recognize a melody in any key. The key a tune is played in refers to the musical scale the melody is built on. The same melody played in different keys starts on different notes. Once you choose the key for a rendition, you\\u2019ve determined the rest of the notes in the tune. Any melody can be played in any key. This \", mdx(\"a\", {\n    id: \"page_81\"\n  }), \"means that each rendition of the \\u201Csame\\u201D melody in a new key is actually an entirely different sequence of notes! Each rendition stimulates an entirely different set of locations on your cochlea, causing an entirely different set of spatial-temporal patterns to stream up into your auditory cortex \\u2026 and yet you perceive the same melody in each case. Unless you have perfect pitch you cannot even distinguish the same song played in two different keys without hearing them back to back.\"), mdx(ContentRef, {\n    id: 46,\n    mdxType: \"ContentRef\"\n  }, \"Think of the song \\u201CSomewhere over the Rainbow.\\u201D You probably first learned it by hearing Judy Garland sing it in the movie \", mdx(\"em\", null, \"The Wizard of Oz,\"), \" but unless you have perfect pitch you probably can\\u2019t recall the key she sang it in (A flat). If I sit down at a piano and start to play the song in a key in which you\\u2019ve never heard it\\u2014say, in D\\u2014it will sound like the same song. You won\\u2019t notice that all the notes are different from those in the version you\\u2019re familiar with. This means that your memory of the song must be in a form that ignores pitch. The memory must store the important relationships in the song, not the actual notes. In this case, the important relationships are the relative pitch of the notes, or \\u201Cintervals.\\u201D \\u201CSomewhere over the Rainbow\\u201D begins with an octave up, followed by a halftone down, followed by a major third down, and so on. The interval structure of the melody is the same for any rendition in any key. Your ability to easily recognize the song in any key indicates that your brain has stored it in this pitch-invariant form.\"), mdx(ContentRef, {\n    id: 47,\n    mdxType: \"ContentRef\"\n  }, \"Similarly, the memory of your friend\\u2019s face must also be stored in a form that is independent of any particular view. What makes her face recognizable are its relative dimensions, relative colors, and relative proportions, not how it appeared one instant last Tuesday at lunch. There are \\u201Cspatial intervals\\u201D between the features of her face just as there are \\u201Cpitch intervals\\u201D between the notes of a song. Her face is wide relative to her eyes. Her nose is short relative to the width of her eyes. The color of her hair and the color of her eyes have a similar relative \", mdx(\"a\", {\n    id: \"page_82\"\n  }), \"relationship that stays constant even though in different lighting conditions their absolute colors change significantly. When you memorized her face, you memorized these relative attributes.\"), mdx(ContentRef, {\n    id: 48,\n    mdxType: \"ContentRef\"\n  }, \"I believe a similar abstraction of form is occurring throughout the cortex, in every region. This is a general property of the neocortex. Memories are stored in a form that captures the essence of relationships, not the details of the moment. When you see, feel, or hear something, the cortex takes the detailed, highly specific input and converts it to an invariant form. It is the invariant form that is stored in memory, and it is the invariant form of each new input pattern that it gets compared to. Memory storage, memory recall, and memory recognition occur at the level of invariant forms. There is no equivalent concept in computers.\"), mdx(ContentRef, {\n    id: 49,\n    mdxType: \"ContentRef\"\n  }, \"This brings up an interesting problem. In the next chapter I argue that an important function of the neocortex is to use its memory to make predictions. But given that the cortex stores invariant forms, how can it make specific predictions? Here are some examples to illustrate the problem and the solution.\"), mdx(ContentRef, {\n    id: 50,\n    mdxType: \"ContentRef\"\n  }, \"Imagine it is 1890, and you are in a frontier town in the American West. Your sweetheart is taking the train from the East to join you in your new frontier home. You of course want to meet her at the station when she arrives. For a few weeks prior to her arrival day you keep track of when the trains come and go. There is no set schedule and as far as you can tell the train never arrives or leaves at the same time during the day. It is beginning to look as if you won\\u2019t be able to predict when her train will arrive. But then you notice there is some structure to the trains\\u2019 comings and goings. The train coming from the East arrives four hours after one leaves heading east. This four-hour gap is consistent day to day although the specific times vary greatly. On the day of her arrival, you keep an eye out for the eastbound \", mdx(\"a\", {\n    id: \"page_83\"\n  }), \"train, and when you see it, you set your clock. After four hours you head for the station and meet her train just as it arrives. This parable illustrates both the problem the neocortex faces and the solution it uses to solve it.\"), mdx(ContentRef, {\n    id: 51,\n    mdxType: \"ContentRef\"\n  }, \"The world as seen by your senses is never the same; like the arrival and departure time of the train, it is always different. The way you understand the world is by finding invariant structure in the constantly changing stream of input. However, this invariant structure alone is not sufficient to use as a basis for making specific predictions. Just knowing that the train arrives four hours after it departs doesn\\u2019t allow you to show up on the platform exactly in time to greet your sweetheart. To make a specific prediction, the brain must combine knowledge of the invariant structure with the most recent details. Predicting the arrival time of the train requires recognizing the four-hour structure in the train schedule, and combining it with the detailed knowledge of what time the last eastbound train left.\"), mdx(ContentRef, {\n    id: 52,\n    mdxType: \"ContentRef\"\n  }, \"When listening to a familiar song played on a piano, your cortex predicts the next note before it is played. But the memory of the song, as we\\u2019ve seen, is in a pitch-invariant form. Your memory tells you what interval is next, but says nothing, in and of itself, about the actual note. To predict the exact next note requires combining the next interval with the last specific note. If the next interval is a major third and the last note you heard was C, then you can predict the specific next note, E. You hear in your mind E, not \\u201Cmajor third.\\u201D And unless you\\u2019ve misidentified the song or the pianist slips up, your prediction is correct.\"), mdx(ContentRef, {\n    id: 53,\n    mdxType: \"ContentRef\"\n  }, \"When you see your friend\\u2019s face, your cortex fills in and predicts the myriad details of her unique image at that instant. It checks that her eyes are just right, that her nose, lips, and hair are exactly as they should be. Your cortex makes these predictions with great specificity. It can predict low-level details about her face even though you have never seen her in this particular orientation or environment before. If you know exactly where \", mdx(\"a\", {\n    id: \"page_84\"\n  }), \"your friend\\u2019s eyes and nose are, and you know the structure of her face, then you can predict exactly where her lips should be. If you know her skin is being tinged orange by the light of sunset, then you know what color her hair should appear. Once again, your brain does this by combining a memory of the invariant structure of her face with the particulars of your immediate experience.\"), mdx(ContentRef, {\n    id: 54,\n    mdxType: \"ContentRef\"\n  }, \"The train schedule example is just an analogy of what is going on in your cortex, but the melody and face examples are not. The combining of invariant representations and current input to make detailed predictions is exactly what is happening. It is a ubiquitous process that happens in every region of cortex. It is how you make specific predictions about the room you are sitting in right now. It is how you are able to predict not only the words others will say, but also in what tone of voice they will say them, the accent they will use, and where in the room you expect to hear the voice come from. It is how you know precisely when your foot will hit the floor, and what it will feel like when you climb a set of stairs. It is how you can sign your name with your foot, or catch a thrown ball.\"), mdx(ContentRef, {\n    id: 55,\n    mdxType: \"ContentRef\"\n  }, \"The three properties of cortical memory discussed in this chapter (storing sequences, auto-associative recall, and invariant representations) are necessary ingredients to predict the future based on memories of the past. In the next chapter I propose that making predictions is the essence of intelligence.\"));\n}\n;\nMDXContent.isMDXComponent = true;","fields":{"slug":"/intelligence/7/"},"frontmatter":{"isBook":false,"title":"4. MEMORY","bookTitle":"On Intelligence","numSections":16,"tags":["a"],"author":"Jeff Hawkins"}}},"pageContext":{"id":"54646f9d-8966-50f2-a205-bd04def20166"}},"staticQueryHashes":["4080856488"]}