Six Areas of Adaptive Neurotechnologies

Introduction

Why do some people need any kind of AN? Most adults can talk, write email, move to different rooms, get a glass of water, or perform other everyday tasks without assistance. But millions of people in the US alone have trouble speaking, walking, using their hands, or performing other tasks that require movement. For some of these people, ANs can replace these lost capabilities by replacing the brain’s normal output pathways. Instead of relying on information sent from the brain to the muscles, ANs can detect the user’s mental activity and send those messages or commands to an artificial device.

Examples of BCIs to replace lost functions

There are many different ways that ANs can replace functions that a user lost due to injury, disease, or other causes. These ways can be viewed as tools for communication or device control.

Communication

One of the most basic communication needs is simply answering questions with “YES” or “NO”. This may seem very limited, but it can make a huge difference in the lives of patients, as well as the people who love them or care for them. Someone might ask questions like “Do you have pain in your left foot?” or “Should I turn on the fan now?” or “Do you want to move to the living room now?” It’s very important to phrase questions carefully so the user can clearly respond with yes or no. Asking a patient if s/he wants to hear music could be confusing. What kind of music? Will you just play the song once, or will you put on background music continuously? Thus, several yes/no questions might be needed to fully determine what the user wants, and people need to make sure the patient can understand the different options that are available. Similarly, asking if a patient would like a chicken enchilada or wants to go swimming can be foolish if the patient can only eat through a feeding tube or has lost the motor control needed to swim. Some ANs have gone beyond yes and no by providing a third option such as “skip”, meaning that the user doesn’t want to answer that question now. Another option is “end”, meaning that the user wants to end the communication session. But, with yes, no, skip, or end, we’re already up to four possible choices that the user could communicate. With only six choices, a user could choose from any letter in the English alphabet or the numbers from 1 to ten. Of course, this requires making two selections. First, the user chooses a group of six items, such as the letters A-H, and then the system could present a second level of choices with those eight letters. Some AN-based spellers use two levels of selection like this. Many systems used for spelling don’t require two levels. For example, the user might see a monitor that has six rows and six columns. You can ask the user to focus on one of these 36 items, and maybe perform a mental task like counting each time a certain letter flashes. ANs could detect the user’s brain activity to determine which of the 36 items s/he wants to communicate. What next? The system might just present answers like yes or no, or more complex messages with spellers, on the monitor. This can be adequate if someone is in the room watching the monitor. However, once you know what someone wants to communicate, it’s relatively easy to convey that message in other ways, such as through a voice synthesizer, email, Twitter, or Facebook. Communication can be more challenging with some users. ANs have been developed for people who aren’t familiar with the English language, such as spellers for Chinese, Japanese, and Arabic speakers. Other ANs have provided communication tools that don’t require vision for people with visual disabilities. Instead of observing items on a monitor, a user might pay attention to different sounds or vibrations, such as stimuli presented to the left vs. right ear or the left vs. right hand. In another approach, people could imagine moving the left hand to convey yes or move a cursor left toward a target letter, or instead imagine moving the right hand to convey no or move a cursor toward a different letter.

Control

ANs can also replace lost functions by helping people control devices, such as a wheelchair, prosthetic arm, or a mobile robot. The mobile robot could go to the kitchen, open the refrigerator, get a glass of water, and bring it to the patient. These control applications have to rely heavily on intelligent software that can manage many complex details that we take for granted. If you want to get a glass of water, you have to figure out the best path to get to the kitchen and back. This requires detecting any obstacles, like chairs or tables, and avoiding them. This may seem easy for people, but robotic devices like wheelchairs and mobile robots need cameras and advanced software to detect and avoid obstacles. Many ANs for control don’t rely on any outside devices beyond a laptop. People could control a web browser or common software applications with AN. Some groups have developed ANs to control games. The commercial game “World of Warcraft” has been used in a few different ANs, since it was a very popular game. In one approach, a user’s character changed from an elf to a bear if the user’s brain activity conveyed stress. Other groups have let users move the game character in three directions, with a fourth option that can perform different tasks depending on the game context. A few companies have released their own games designed to work with BCIs. Ultimately, we’d like to provide ANs that can replace lost functions as completely as possible – meaning that the user can just perform the same mental activities as a healthy person, and get the same results. This is a long way off. Communication with ANs is much slower than typing or talking. Controlling all of the details of a prosthetic hand, including each finger movement, is not yet possible. Moving a wheelchair or robot to another room takes a lot more time than just walking there. However, there has been a lot of progress. The last few years have seen better sensors to detect brain activity, improved interaction with other software, immersive VR-based environments that are designed to be as user-friendly as possible, faster recognition of what the user wants to convey, and improved customization to each individual user. We hope that ongoing research will lead to new ANs that can help meet the needs and desires of the many different people who need help to replace lost functions.

Introduction

Many people have trouble doing things that were once easy. For example, some people may have intact arms or legs, but can no longer use them because of damage to the brain or spinal cord. Difficulty controlling bladder or bowel activity can also occur. Some ANs can help restore lost functions by sending information directly from the brain to devices that can help their arms, legs, or other areas work properly.

Examples of ANs to restore lost functions

Movement: Imagination to reality

For most people, moving the hand is easy. You just think about grasping with your right hand, and it moves just the way you think. But what if the brain areas that produce that movement are damaged? What if the brain is intact, but the spinal cord can no longer send information from the brain to the hand? For these users, and some others with movement disabilities, an orthosis could be the best choice to restore control of movement. An orthosis is a device that can be attached to part of the body, such as a hand, to help that body part complete a movement.
Muscle control can also be restored with a device called a functional electrical stimulator, or FES. An FES is a device that sends a weak electrical current through the muscles. This electrical current can make the muscles activate. This is similar to how your nervous system naturally controls muscles. People who want to move usually send a signal from the brain, along the spinal cord, that causes electrical activity to trigger movement. With FES, this electrical activity comes from an outside device instead. FES pads must be placed over specific muscles that are responsible for a movement. For example, placing FES pads placed over the forearm could cause the hand to flex at the wrist.
These types of devices can be controlled with ANs that directly read brain activity relating to movement. When you think about grasping with your right hand, the motor areas of your brain are active. In fact, millions of neurons work together for even simple movements. This activity varies a little across different people, and ANs that use motor imagery may need some time to learn each person’s unique brain signals. But, the brain activity involved in different kinds of movements is generally predictable. So, with a few hours of training, people could control an orthosis or FES that makes their right hand grasp or open just by thinking about one of those two movements.
While grasping is a common task that ANs can help restore, some people need other types of ANs to control other parts of the body. Some ANs are designed to help people who have trouble walking by stimulating muscles in the feet, legs, and hips. ANs can also control exoskeletons and/or orthoses that influence these parts of the body. The basic principle is the same: when you think about walking, the movement areas of the brain are active in relatively predictable ways. Thinking about walking doesn’t produce exactly the same activity as thinking about moving the hand, so ANs could potentially be used to restore function in different parts of the body.
Besides restoring the ability to walk, ANs can also help people with gait. Many people can walk, but have trouble walking smoothly. One major cause is Parkinson’s disease. People with this disease may still be able to walk, but their gait can appear jerky and erratic, because the brain areas that control movement no longer produce the chemicals needed for smooth movements. For these people, walking can be risky, with a high risk of falling. Besides, many patients are understandably uncomfortable walking or performing other movements when other people can see that they have a problem. Aside from restoring basic functionality and safety, ANs can help restore dignity to people who might be embarrassed to invite friends to their homes or go outside. ANs that help restore the ability to walk can be more challenging than ANs for grasping for two reasons. First, ANs for walking are usually more bulky and expensive, because walking requires precise control of many different muscle groups. Second, ANs for walking involve more issues with safety, since mistakes could cause the user to fall.

Bladder and bowel control

Many persons with disabilities cannot control the muscles that are responsible for bladder and bowel control. ANs could be used to activate a device that activates the bladder or bowel muscles based on brain activity. This would not require high-bandwidth communication, since these are activities that people select relatively infrequently, and can be executed with a simple switch mechanism.

Introduction

Most ANs are designed to provide some benefit very quickly, as part of a real-time interaction. If a user wants to stand up with the help of AN, a delay of even a second between thought and action can be frustrating. ANs for communication, gaming, virtual navigation, web browsing, and many other applications typically focus on accomplishing a user’s immediate goals as quickly as possible. If the user wants some benefit from the AN, then the AN has to be active at that moment to provide that benefit.
However, ANs to improve function instead aim to create a lasting improvement in the nervous system – changes that persist even when the AN is no longer being used. Hundreds of millions of people have trouble with movement, memory, attention, or other mental activities. These motor and cognitive impairments could result from stroke, injuries, dementia, Alzheimer’s disease, Parkinson’s disease, or other causes. ANs could help people retrain their nervous systems to counteract damage, leaving them with an improved capability to function even when they are not using ANs anymore. These types of ANs could help healthy people as well, along with ANs to reduce stress.

Examples of ANs to improve functions

Rehabilitation

Some ANs aim to help people re-train the brain or spinal cord to work properly again. Millions of people each year participate in physiotherapy, which means that a medical expert called a physiotherapist helps them perform exercises that are designed to help them learn to move again. Often, this physiotherapy involves a device called a functional electrical stimulator, or FES, described in the “restore” section. Physiotherapists may also use a virtual avatar that moves the hand when the user is supposed to move the hand.
In the last few years, several groups have described ANs that could provide better results by linking the FES system and other tools to brain activity. That is, the system only activates the FES or the avatar when AN detects the correct movement imagination in the user’s brain. This helps patients learn how to imagine (and then perform) the movement again. Tools like orthoses and exoskeletons can also be important components of a rehabilitation system.
Another option for rehabilitation involves the H-reflex. The H-reflex is a natural reaction that people have after certain types of muscle stimulation. ANs can use the H-reflex to help restore spinal cord function that they lost because of an injury. This approach has already shown remarkable results in testing with rats and cats, leading to high hopes for extending this work to help people.
ANs are helpful for people with trouble moving and also for people with other problems. Some groups have developed ANs to help people improve their memory. In this approach, the ANs can detect brain activity related to mental tasks like remembering cards or letters. The AN might show people a sequence of letters and then ask: is this the same as the letter that you saw earlier? Like other ANs for rehabilitation, these tools can help people restore neurons involved in memory. Related work could help people who have trouble with attention, emotion, or other mental functions.

Healthy users

These tools to improve function are also being explored for healthy users. Some groups have studied ANs with elite competitive athletes in several different sports. For these users, even tiny improvements could make the difference in winning a competition. More broadly, many people would love to be faster, stronger, nimbler, or smarter. Future ANs might be able to help with these goals, but raise some ethical issues that should be considered further.
In addition to improving peak performance, ANs could also reduce the time needed to learn new skills. Research has provided the theoretical basis for learning different types of skills and facts more quickly. People could use this to improve their work productivity or performance, or to learn new games or hobbies. This is also a more futuristic direction for ANs.
One application for healthy users is not especially futuristic – relaxation. For decades, people have used different systems that can help them relax. Here, the goal is to provide some improvement that persists when the AN is no longer active, but the improvement is simply feeling more peaceful. Older systems detected tension from electrical activity in the brain and muscles. Newer tools can also use methods like eye tracking and posture analysis. By helping people reduce activities like jaw clenching, teeth grinding, forehead tension, poor posture, and rapid shifts of attention, ANs can improve well-being.

Introduction

ANs can also help enhance CNS function. While our brains are capable of some remarkable tasks, we can’t always do them perfectly. People might make mistakes because they are tired, inexperienced, overloaded, or confused. Even experts who feel they’re ready for challenging tasks make mistakes. Drugs, alcohol, injuries, illness, distraction, and other factors can also reduce performance. Some types of ANS can help detect and counteract problems that result from various causes of human error.

Examples of BCIs to enhance functions

Alertness

Nobody can be alert all the time. People can recognize some times when they might not be at their best, such as when they feel sleepy or have been performing a boring task for a long time. Research has also shown that people may be prone to mistakes even when they feel alert. People who doze off when driving – even for a second – can cause accidents. Other people, such as pilots, train conductors, surgeons, security staff, or some military personnel can also create serious risks when their brains are not functioning at peak performance.
One very well-established direction for ANs involves monitoring alertness. When people become tired, they have predictable changes that can be detected. You’re probably familiar with the feeling of heavy eyelids when you are tired, and devices to measure eye activity can be a good way to monitor alertness. EEG and heart activity can also indicate lapses in alertness and can help identify periods when errors are more likely.
For example, some car companies have been exploring ways to detect alertness lapses in drivers. NASA and other groups have long been exploring ways to use ANs to determine when pilots, astronauts, or others might not be able to work effectively. Research from about 20 years ago studied expert sonar operators performing realistic sonar detection task. The authors found that these sonar operators were prone to make mistakes at certain times when their EEG showed predictable patterns.

Overload

The most alert people can still make mistakes if they can’t manage all of the mental tasks expected of them. This can be especially problematic if people are confused or don’t fully understand what they are supposed to do. Often, people might not want to say they don’t know what to do or can’t handle work because they do not want to look stupid or get in trouble. Different groups have studied how the EEG and other signals change when people are overloaded with too much information and/or too many tasks based on this information. In a typical study, users are asked to perform a task that is simple at first, but becomes more difficult. When people start to miss information presented on a monitor, fail to press the right buttons, or simply say they’ve had too much, researchers can study what changed through AN.

Error detection

A closely related direction involves using the EEG and other activity to determine when people think they just made a mistake. This approach can’t identify all mistakes, since it relies on people recognizing that they made a mistake. Still, this is a fairly common situation. Some work has proposed an automated backspace feature if someone errs while spelling.
What can be done with this information? This is a question that goes beyond ANs, and depends heavily on the situation. ANs might simply inform the user that he or she should rest. ANs could also inform a supervisor, who might ask the user to take a break, review recently completed tasks, or ask another employee to take over. Automated tools could stop processing work that the user just completed, flag it for later review, offload some tasks onto another user or a software routine, or attempt error correction.
More broadly, information from monitoring alertness, overload, or errors could be used to learn more about different software, industrial designs, work assignments, or other details help people perform most effectively. These non-realtime methods are related to human factors engineering human-computer interaction (HCI), two fields that are concerned with finding the most efficient, usable ways to help people interact with machines.

Training and optimization

ANs have also been used to study how people learn new tasks and perform them most effectively. Employers (including the private and military sectors) have to spend a lot of time and money training new people. Aside from work, people often want to be better at sports, musical instruments, games, or other activities. Elite athletes are interested in even tiny improvements that could give them an edge. ANs can use different signals to learn how the CNS changes while learning and performing tasks to reduce training time and maximize performance.

Object detection

There are some situations that require people to detect certain target items and then press a button or perform some other tasks when they appear. For example, staff might be asked to view thousands of pictures from a traffic camera and identify one truck with a certain license plate or a suspicious feature. ANs have been used to enhance performance by detecting specific brain patterns that reflect that someone just saw a target. This approach can make it possible to present more images per second than waiting for someone to press a button. The system can then identify which pictures got the user’s attention and present them for later review while discarding the pictures that were not considered targets.

Introduction

“I only have two hands!” This common expression underscores that people often wish they could supplement their natural abilities with something like a third hand. These kinds of ANs have not been explored as much as other directions, and supplemental ANs that really provide an overall boost to performance are somewhat further in the future than other ANs.

Examples of ANs to supplement functions

Third hand/arm

As we discussed elsewhere on this web page, ANs can be used to help control a patient’s arm or a prosthetic arm. In addition to helping patients get closer to the natural human capacity to use two arms, these types of ANs could also provide control of one or more additional arms. People could type with both hands while also using a pencil, cell phone, remote control, or musical instrument. In addition to providing the ability to do different conventional tasks at the same time, ANs could provide new capabilities, like carrying bulkier objects or playing more complex musical pieces. More elaborate ANs could even provide control of an exoskeleton that could supplement or improve our bodies in different ways.

Extra finger

For more precise control, one extra finger might let people type faster or play games more competitively. Even an additional binary command could allow an extra “select” or “confirm” command to improve human-computer interaction. This might be possible through ANs that can directly or indirectly simulate a mouse click, keystroke, or button press. An extra finger might supplement non-electronic activities that require complex finger movements like knitting, gardening, or washing dishes.

Artistic expression

People can already play musical instruments, dance, or act. But what if ANs could supplement these natural types of artistic expression? For example, some groups have proposed or demonstrated ANs that create a new type of music. They combine music from people playing classical instruments with computer-generated music based on a user’s mood derived from brainwaves. ANs could also measure brain activity to change background lighting, costumes, or other aspects of performance art.

Challenges with ANs to supplement functions

From the standpoint of the signals recorded from the brain, ANs for supplemental control could be quite different from ANs to restore, improve, or replace control of peoples’ natural limbs. ANs based on motor imagery typically rely on the signals that the brain naturally produces when imagining movement. However, people don’t naturally produce brain signals to control a third hand or eleventh finger. Perhaps people could learn to control a supplemental arm with the same signals that control a natural arm, and/or learn to perform other tasks that can provide supplemental control. Or, like the ANs for musical expression, ANs could supplement activities with naturally occurring brain signals.
One issue that is especially challenging in ANs to supplement function is distraction. The user may have to devote some attention to control the AN as well as another task without getting too distracted. Depending on the overall system design, the user may need to do many things simultaneously. If this multitasking gets too confusing or stressful, then the supplemental AN wouldn’t be very helpful overall. In some domains, such as computer games that require very fast reflexes, users would probably find that using AN while playing a game is so distracting that their game performance declines. Further research is needed to develop task combinations, usage environments, and AN systems that are seamless and easy to use. Distraction may be less of a problem when the AN doesn’t require performing additional mental tasks to control it.

A wide variety of artificial neural networks are being used to try to duplicate and elucidate central nervous system (CNS) function. AN-based lab models that interact directly with the neurons of actual CNS networks, while not perfect surrogates for natural behaviors, avoid the complications of the musculoskeletal apparatus and its complex sensory feedback. Insights from these AN-based models, which incorporate actual CNS neurons and their connections, are likely to be more relevant to CNS function than those gleaned from entirely artificial neural network models. For example, AN-based models can compare the impact on an already acquired behavior of the acquisition of a new behavior that lies entirely within the current intrinsic manifold (i.e., already available combinations of active single neurons) to the impact of a new behavior that requires combinations not within the manifold (Oby et al, PNAS 116:15210-15215, 2019; Wolpaw & Kamesar, J Physiol 600.15, 3423-2452, 2022).