The authors would like to thank all the participants for their time and involvement.

The current study aims to investigate the feasibility of controlling a 3D avatar in real-time within a VR environment using MI signals recorded via EEG. The study focuses on enhancing immersion and the sense of body ownership by personalizing the avatar to resemble the subject closely. The protocol received approval from the Vellore Institute of Technology Review Board. Participants provided written informed consent after reviewing the study&#39;s purpose, procedures, and potential risks.
<stron...

Motor Imagery Using VR Digital Avatars in a Brain-Computer Interface System

<ol>
	<li>Andrade, J., Cec&#237;lio, J., Sim&#245;es, M., Sales, F., Castelo-Branco, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Separability+of+motor+imagery+of+the+self+from+interpretation+of+motor+intentions+of+others+at+the+single+trial+level:+An+eeg+study.">Separability of motor imagery of the self from interpretation of motor intentions of others at the single trial level: An eeg study.</a> J. NeuroEng. Rehabil. 14 (1), 1-13 (2017).</li><li>Lorey, B., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Neural+simulation+of+actions:+Effector-versus+action-specific+motor+maps+within+the+human+premotor+and+posterior+parietal+area.">Neural simulation of actions: Effector-versus action-specific motor maps within the human premotor and posterior parietal area.</a> Hum. Brain Mapp. 35 (4), 1212-1225 (2014).</li><li>Ehrsson, H. H., Geyer, S., Naito, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Imagery+of+voluntary+movement+of+fingers,+toes,+and+tongue+activates+corresponding+body-part-specific+motor+representations.">Imagery of voluntary movement of fingers, toes, and tongue activates corresponding body-part-specific motor representations.</a> J Neurophysiol. 90 (5), 3304-3316 (2003).</li><li>Sauvage, C., Jissendi, P., Seignan, S., Manto, M., Habas, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Brain+areas+involved+in+the+control+of+speed+during+a+motor+sequence+of+the+foot:+Real+movement+versus+mental+imagery.">Brain areas involved in the control of speed during a motor sequence of the foot: Real movement versus mental imagery.</a> J Neuroradiol. 40 (4), 267-280 (2013).</li><li>Pfurtscheller, G., Neuper, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Motor+imagery+activates+primary+sensorimotor+area+in+humans.">Motor imagery activates primary sensorimotor area in humans.</a> Neurosci Lett. 239 (2-3), 65-68 (1997).</li><li>Jeon, Y., Nam, C. S., Kim, Y. J., Whang, M. C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Event-related+(de)synchronization+(erd/ers)+during+motor+imagery+tasks:+Implications+for+brain-computer+interfaces.">Event-related (de)synchronization (erd/ers) during motor imagery tasks: Implications for brain-computer interfaces.</a> Int. J. Ind. Ergon. 41 (5), 428-436 (2011).</li><li>Mcfarland, D. J., Miner, L. A., Vaughan, T. M., Wolpaw, J. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Mu+and+beta+rhythm+topographies+during+motor+imagery+and+actual+movements.">Mu and beta rhythm topographies during motor imagery and actual movements.</a> Brain Topogr. 12 (3), 177-186 (2000).</li><li>Di Pellegrino, G., Fadiga, L., Fogassi, L., Gallese, V., Rizzolatti, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Understanding+motor+events:+A+neurophysiological+study.">Understanding motor events: A neurophysiological study.</a> Exp Brain Res. 91 (1), 176-180 (1992).</li><li>Rizzolatti, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+mirror+neuron+system+and+its+function+in+humans.">The mirror neuron system and its function in humans.</a> Anat Embryol (Berl). 210 (5-6), 419-421 (2005).</li><li>Jackson, P. L., Lafleur, M. F., Malouin, F., Richards, C. L., Doyon, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Functional+cerebral+reorganization+following+motor+sequence+learning+through+mental+practice+with+motor+imagery.">Functional cerebral reorganization following motor sequence learning through mental practice with motor imagery.</a> Neuroimage. 20 (2), 1171-1180 (2003).</li><li>Cramer, S. C., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Harnessing+neuroplasticity+for+clinical+applications.">Harnessing neuroplasticity for clinical applications.</a> Brain. 134, 1591-1609 (2011).</li><li>Nojima, I., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Human+motor+plasticity+induced+by+mirror+visual+feedback).">Human motor plasticity induced by mirror visual feedback).</a> J Neurosci. 32 (4), 1293-1300 (2012).</li><li>Slater, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Implicit learning through embodiment in immersive virtual reality. , (2017).</li><li>Tham, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Understanding+virtual+reality:+Presence,+embodiment,+and+professional+practice.">Understanding virtual reality: Presence, embodiment, and professional practice.</a> IEEE Trans. Prof. Commun. 61 (2), 178-195 (2018).</li><li>Choi, J. W., Kim, B. H., Huh, S., Jo, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Observing+actions+through+immersive+virtual+reality+enhances+motor+imagery+training.">Observing actions through immersive virtual reality enhances motor imagery training.</a> IEEE Trans. Neural Syst. Rehabil. Eng. 28 (7), 1614-1622 (2020).</li><li>Lakshminarayanan, K., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+effect+of+combining+action+observation+in+virtual+reality+with+kinesthetic+motor+imagery+on+cortical+activity.">The effect of combining action observation in virtual reality with kinesthetic motor imagery on cortical activity.</a> Front Neurosci. 17, 1201865 (2023).</li><li>Juliano, J. M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Embodiment+is+related+to+better+performance+on+a+brain-computer+interface+in+immersive+virtual+reality:+A+pilot+study.">Embodiment is related to better performance on a brain-computer interface in immersive virtual reality: A pilot study.</a> Sensors. 20 (4), 1204 (2020).</li><li>Lakshminarayanan, K., Shah, R., Yao, Y., Madathil, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+effects+of+subthreshold+vibratory+noise+on+cortical+activity+during+motor+imagery.">The effects of subthreshold vibratory noise on cortical activity during motor imagery.</a> Motor Control. 27 (3), 559-572 (2023).</li><li>Cole, S. W., Yoo, D. J., Knutson, B. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Interactivity+and+reward-related+neural+activation+during+a+serious+videogame.">Interactivity and reward-related neural activation during a serious videogame.</a> PLoS One. 7 (3), e33909 (2012).</li><li>Cameirao, M. S., Badia, S. B., Duarte, E., Frisoli, A., Verschure, P. F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+combined+impact+of+virtual+reality+neurorehabilitation+and+its+interfaces+on+upper+extremity+functional+recovery+in+patients+with+chronic+stroke.">The combined impact of virtual reality neurorehabilitation and its interfaces on upper extremity functional recovery in patients with chronic stroke.</a> Stroke. 43 (10), 2720-2728 (2012).</li><li>Turolla, A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Virtual+reality+for+the+rehabilitation+of+the+upper+limb+motor+function+after+stroke:+A+prospective+controlled+trial.">Virtual reality for the rehabilitation of the upper limb motor function after stroke: A prospective controlled trial.</a> J Neuroeng Rehabil. 10, 85 (2013).</li><li>Isaacson, B. M., Swanson, T. M., Pasquina, P. F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+use+of+a+computer-assisted+rehabilitation+environment+(caren)+for+enhancing+wounded+warrior+rehabilitation+regimens.">The use of a computer-assisted rehabilitation environment (caren) for enhancing wounded warrior rehabilitation regimens.</a> J Spinal Cord Med. 36 (4), 296-299 (2013).</li><li>Alchalabi, B., Faubert, J., Labbe, D. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=EEG+can+be+used+to+measure+embodiment+when+controlling+a+walking+self-avatar.">EEG can be used to measure embodiment when controlling a walking self-avatar.</a> IEEE Conf. on Virtual Reality and 3D User Interfaces (VR). , 776-783 (2019).</li><li>Luu, T. P., Nakagome, S., He, Y., Contreras-Vidal, J. L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Real-time+eeg-based+brain-computer+interface+to+a+virtual+avatar+enhances+cortical+involvement+in+human+treadmill+walking.">Real-time eeg-based brain-computer interface to a virtual avatar enhances cortical involvement in human treadmill walking.</a> Sci Rep. 7 (1), 8895 (2017).</li><li>Longo, B. B., Benevides, A. B., Castillo, J., Bastos-Filho, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Using+brain-computer+interface+to+control+an+avatar+in+a+virtual+reality+environment.">Using brain-computer interface to control an avatar in a virtual reality environment.</a> 5th ISSNIP-IEEE Biosignals and Biorobotics Conf. (BRC). , 1-4 (2014).</li><li>Hinrichs, H., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Comparison+between+a+wireless+dry+electrode+eeg+system+with+a+conventional+wired+wet+electrode+eeg+system+for+clinical+applications.">Comparison between a wireless dry electrode eeg system with a conventional wired wet electrode eeg system for clinical applications.</a> Scientific reports. 10 (1), 5218 (2020).</li><li>&#352;kola, F., Tinkov&#225;, S., Liarokapis, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Progressive+training+for+motor+imagery+brain-computer+interfaces+using+gamification+and+virtual+reality+embodiment.">Progressive training for motor imagery brain-computer interfaces using gamification and virtual reality embodiment.</a> Front Human Neurosci. 13, 329 (2019).</li></ol>

The application of MI in conjunction with VR technology offers a promising avenue for rehabilitation by leveraging the brain&#39;s natural mechanisms for motor planning and execution. MI&#39;s ability to induce event-related desynchronization in specific brain frequency bands, mirroring the neural activity of physical movement2,3,4, provides a robust framework for engaging and strengthening the neural networks involved in motor ...

Rehabilitation paradigms for patients with neurological impairments are undergoing a transformative shift with the integration of advanced technologies such as brain-computer interfaces (BCI) and immersive virtual reality (VR), offering a more nuanced and effective method for fostering recovery. Motor imagery (MI), the technique at the heart of BCI-based rehabilitation, involves the mental rehearsal of physical movements without actual motor execution1. MI exploits a neural mechanism where imagining a movement triggers a pattern of brain activity that closely mirrors that of performing the physical action itself2,3,4. Specifically, engaging in MI leads to a phenomenon known as event-related desynchronization (ERD) in the alpha (8-13 Hz) and beta (13-25 Hz) frequency bands of the brain&#39;s electrical activity5,6,7. ERD is indicative of a suppression of the baseline brain rhythms, a pattern also observed during actual movement, thereby providing a neural substrate for the use of MI within BCI-assisted rehabilitation frameworks7. Such a similarity in cortical activation between MI and physical movement suggests that MI can effectively stimulate the neural networks involved in motor control, making it a valuable tool for patients with motor deficits8. Furthermore, the practice of MI has been extended beyond mere mental rehearsal to include action observation strategies9. Observing the movement of task-related body parts or actions in others can activate the mirror neuron network (MNN), a group of neurons that respond both to action observation and execution9. Activation of the MNN through observation has been demonstrated to induce cortical plasticity, as evidenced by various neuroimaging modalities, including functional MRI10, positron emission tomography11, and transcranial magnetic stimulation12. The evidence supports the notion that MI training, enhanced by action observation, can lead to significant neural adaptation and recovery in affected individuals.
Virtual reality technology has revolutionized the realm of MI-based rehabilitation by offering an immersive environment that enhances the sense of body ownership and blurs the distinctions between the real and virtual worlds13,14,15. The immersive quality of VR makes it an effective tool for action observation and motor imagery practice, as it allows participants to perceive the virtual environment as real15. Research has shown that VR devices have a more pronounced effect on MI training compared to traditional 2D monitor displays15,16. Such findings are evidenced by enhanced neural activity, such as increased ERD amplitude ratios in the sensorimotor cortex, highlighting the benefits of higher immersion levels in stimulating brain activity during visually guided MI exercises16. The system aids in improving MI performance for tasks involving arm or limb movements by providing direct feedback, thereby enhancing the rehabilitation process16,17. The synergy between MI and VR emphasizes integrating sensory, perceptual, cognitive, and motor activities18,19. The combination has been particularly beneficial for stroke survivors20,21 and war veterans22, as studies have shown that integrating VR into MI-based rehabilitation protocols can significantly reduce rehabilitation time and improve recovery outcomes. The unique feature of VR in rehabilitation lies in its ability to create a sense of presence within a specifically designed virtual environment, enhancing the rehabilitation experience that is further augmented by the inclusion of virtual avatars representing the user&#39;s body, which has been increasingly utilized in motor rehabilitation studies23. These avatars offer a realistic three-dimensional representation of limb movements, aiding in MI and significantly impacting motor cortex activation. By allowing participants to visualize their virtual selves performing specific tasks, VR not only enriches the MI experience but also fosters a more rapid and effective neural reorganization and recovery process24. The implementation of virtual avatars and simulated environments in MI training emphasizes the natural and integrated use of virtual bodies within immersive virtual worlds.
Despite the remarkable advantages of BCI-based control of 3D avatars in MI for rehabilitation, a significant limitation remains in the predominant use of offline methodologies. Currently, most BCI applications involve capturing pre-recorded electroencephalography (EEG) data that is subsequently utilized to manipulate an avatar24,25. Even in scenarios where real-time avatar control is achieved, these avatars are often generic and do not resemble the participants they represent23. This generic approach misses a critical opportunity to deepen the immersion and sense of body ownership, which is crucial for effective rehabilitation24. The creation of a 3D avatar that mirrors the exact likeness of the subject could significantly enhance the immersive experience of the experience16. By visualizing themselves in the virtual world, participants could foster a stronger connection between their imagined and actual movements, potentially leading to more pronounced ERD patterns and, thus, more effective neural adaptation and recovery16. By advancing towards real-time control of personalized 3D avatars, the field of BCI and VR can significantly improve rehabilitation paradigms, offering a more nuanced, engaging, and efficacious method for patient recovery.
The current manuscript presents the creation, design, and technological aspects of both hardware and software of the VR-based real-time BCI control of 3D avatars, highlighting its innovative results that support its integration into motor rehabilitation settings. The proposed system will utilize electroencephalography (EEG) to capture motor imagery signals generated by the subject, which will then be used to control the movements and actions of the avatar in real time. The current approach will combine the advanced capabilities of VR technology with the precision of EEG in recognizing and interpreting brain activity related to imagined movements, aiming to create a more engaging and effective interface for users to interact with digital environments through the power of their thoughts.

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>Alienware Laptop</td>
			<td>Dell</td>
			<td></td>
			<td>High-end gaming laptop with GTX1070 Graphics Card</td>
		</tr>
		<tr>
			<td>Oculus Rift-S VR headset</td>
			<td>Meta</td>
			<td></td>
			<td>VR headset</td>
		</tr>
		<tr>
			<td>OpenBCI Cyton Daisy</td>
			<td>OpenBCI</td>
			<td></td>
			<td>EEG system</td>
		</tr>
		<tr>
			<td>OpenBCI Gel-free cap</td>
			<td>OpenBCI</td>
			<td></td>
			<td>Gel-free cap for placing the EEG electrodes over the participant&#39;s scalp</td>
		</tr>
	</tbody>
</table>

motor imagery performance through embodied digital twins in a virtual reality-enabled brain-computer interface environment

This study introduces an innovative framework for neurological rehabilitation by integrating brain-computer interfaces (BCI) and virtual reality (VR) technologies with the customization of three-dimensional (3D) avatars. Traditional approaches to rehabilitation often fail to fully engage patients, primarily due to their inability to provide a deeply immersive and interactive experience. This research endeavors to fill this gap by utilizing motor imagery (MI) techniques, where participants visualize physical movements without actual execution. This method capitalizes on the brain's neural mechanisms, activating areas involved in movement execution when imagining movements, thereby facilitating the recovery process. The integration of VR's immersive capabilities with the precision of electroencephalography (EEG) to capture and interpret brain activity associated with imagined movements forms the core of this system. Digital Twins in the form of personalized 3D avatars are employed to significantly enhance the sense of immersion within the virtual environment. This heightened sense of embodiment is crucial for effective rehabilitation, aiming to bolster the connection between the patient and their virtual counterpart. By doing so, the system not only aims to improve motor imagery performance but also seeks to provide a more engaging and efficacious rehabilitation experience. Through the real-time application of BCI, the system allows for the direct translation of imagined movements into virtual actions performed by the 3D avatar, offering immediate feedback to the user. This feedback loop is essential for reinforcing the neural pathways involved in motor control and recovery. The ultimate goal of the developed system is to significantly enhance the effectiveness of motor imagery exercises by making them more interactive and responsive to the user's cognitive processes, thereby paving a new path in the field of neurological rehabilitation.

Author Spotlight: Enhancing Neurorehabilitation Through EEG, Motor Imagery, and Virtual Reality

Motor Imagery Performance Through Embodied Digital Twins in a Virtual Reality-Enabled Brain-Computer Interface Environment

My research in neuro rehabilitation centers around improving upper limb function for individuals with neurological injuries or disorders using a combination of EEG, motor imagery, and virtual reality technologies. The primary goal is to understand how these technologies can be integrated to enhance the effectiveness of motor skills rehabilitation. Virtual reality environments have become more sophisticated, enabling realistic simulations, tailored to individual rehabilitation needs, which help in refining motor skills in a nuanced manner.Hybrid systems that combine VR with other technologies like robotic arms and haptic feedback devices are also on the rise. One of the current experimental challenges with the use of brain computer interfaces revolves around the issue of intersubject variability and the need for extensive training. Each individual's brain signal can be vastly different, which means that BCIs often need to be extensively customized or calibrated for each user.Current treatments, often like the immersion and interactivity necessary for maximum efficacy. To fill this gap, our protocol utilizes motor imagery with an innovative twist, integrating digital twins represented by personalized 3D avatars in a virtual reality setting. This integration enhances immersion, making the rehabilitation process not just a mental exercise but also an engaging experience.Our research protocol offers significant advantages over other techniques, particularly in terms of ease of set and cost effectiveness. A standout feature is the creation and utilization of 3D avatars that closely resemble the subjects. This is achieved using simple, readily available tools and software that can generate personalized avatars from basic input data such as photographs.

The results shown are from 5 individuals who followed the protocol described above. A total of 5 healthy adults (3 females) with ages ranging from 21 to 38 years participated in the study.
The individual classification performance for each participant under both motor imagery training and testing conditions is shown in Figure 2. An average confusion matrix for all subjects was calculated to evaluate the classifier&#39;s accuracy in distinguishing between left and ...

Read this Scientific Article Protocol about Motor Imagery Performance through Embodied Digital Twins in a Virtual Reality-Enabled Brain-Computer Interface Environment at JoVE.com

Motor imagery in a virtual reality environment has wide applications in brain-computer interface systems. This manuscript outlines the use of personalized digital avatars that resemble the participants performing movements imagined by the participant in a virtual reality environment to enhance immersion and a sense of body ownership.

This study introduces an innovative framework for neurological rehabilitation by integrating brain-computer interfaces (BCI) and virtual reality (VR) ...

motor-imagery-performance-through-embodied-digital-twins-virtual