Name: ssvep-based experimental procedure for brain-robot interaction with humanoid robots
Uploaded: 2015-11-24T15:00:00
Description: Watch this Scientific Journal Video about SSVEP-based Experimental Procedure for Brain-Robot Interaction with Humanoid Robots at JoVE.com

The authors would like to express their gratitude to Mr. Hong Hu for his assistance in performing the experiments reported in this paper.&#160;&#160;This work was supported in part by the National Natural Science Foundation of China (No. 61473207).

1. Acquiring EEG Signals
<ol>
	<li>Explain the experimental procedure to the subject and obtain written informed consent to participate in experiments.</li>
	<li>Measure the circumference of the subject&#39;s head using a tape measure and select the EEG cap size that is close to the measurement. The electrodes arrangement is based on the &#34;International 10-20 System&#34; 12.</li>
	<li>Measure the distance between the nasion and inion. Use a skin marker pencil to mark 10% of the distance as a reference for...

<ol>
	<li>McFarland, D. J., Wolpaw, J. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Brain-Computer+Interface+Operation+of+Robotic+and+Prosthetic+Devices.">Brain-Computer Interface Operation of Robotic and Prosthetic Devices.</a> Computer. 41, 52-56 (2008).</li><li>Lebedev, M. A., Nicolelis, M. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Brain-machine+interfaces:+Past,+present+and+future.">Brain-machine interfaces: Past, present and future.</a> Trends Neruosci. 29 (9), 536-546 (2006).</li><li>Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., Vaughan, T. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Brain-computer+interfaces+for+communication+and+control.">Brain-computer interfaces for communication and control.</a> Clin. Neurophysiol. 113, 767-791 (2002).</li><li>Bell, C. J., Shenoy, P., Chalodhorn, R., Rao, R. P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Control+of+a+humanoid+robot+by+a+noninvasive+brain-computer+interface+in+humans.">Control of a humanoid robot by a noninvasive brain-computer interface in humans.</a> J. Neural. Eng. 5, 214-220 (2008).</li><li>Li, W., Li, M., Zhao, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Control+of+humanoid+robot+via+motion-onset+visual+evoked+potentials.">Control of humanoid robot via motion-onset visual evoked potentials.</a> Front. Syst. Neurosci. 8, 247 (2014).</li><li>Regan, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Some+characteristics+of+average+steady-state+and+transient+responses+evoked+by+modulated+light.">Some characteristics of average steady-state and transient responses evoked by modulated light.</a> Electroencephalogr. Clin. Neurophysiol. 20, 238-248 (1966).</li><li>Vialatte, F. B., Maurice, M., Dauwels, J., Cichocki, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Steady-state+visually+evoked+potentials:+focus+on+essential+paradigms+and+future+perspectives.">Steady-state visually evoked potentials: focus on essential paradigms and future perspectives.</a> Prog. Neurobiol. 90, 418-438 (2010).</li><li>Bin, G., Gao, X., Wang, Y., Li, Y., Hong, B., Gao, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+high-speed+BCI+based+on+code+modulation+VEP.">A high-speed BCI based on code modulation VEP.</a> J. Neural. Eng. 8, 025015 (2011).</li><li>Sutton, S., Braren, M., Zubin, J., John, E. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Evoked-potential+correlates+of+stimulus+uncertainty.">Evoked-potential correlates of stimulus uncertainty.</a> Science. 150, 1187-1188 (1965).</li><li>Pfurtscheller, G., Lopes da Silva, H. F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Event-related+EEG/MEG+synchronization+and+desynchronization:+basic+principles.">Event-related EEG/MEG synchronization and desynchronization: basic principles.</a> Clin. Neurophysiol. 110, 1842-1857 (1999).</li><li>Zhao, J., Meng, Q., Li, W., Li, M., Sun, F., Chen, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=OpenViBE-based+brainwave+control+system+for+Cerebot.">OpenViBE-based brainwave control system for Cerebot.</a> Proc. IEEE International Conference on Robotics and Biomimetics. , 1169-1174 (2013).</li><li>Homan, R. W., Herman, J., Purdy, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Cerebral+location+of+international+10-20+system+electrode+placement.">Cerebral location of international 10-20 system electrode placement.</a> Electroencephalogr. Clin. Neurophysiol. 66, 376-382 (1987).</li><li>Bin, G., Gao, X., Yan, Z., Hong, B., Gao, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=An+online+multi-channel+SSVEP-based+brain-computer+interface+using+a+canonical+correlation+analysis+method.">An online multi-channel SSVEP-based brain-computer interface using a canonical correlation analysis method.</a> J. Neural. Eng. 6, 046002 (2009).</li><li>Wang, Y., Wang, R., Gao, X., Hong, B., Gao, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+practical+VEP-based+brain-computer+interface.">A practical VEP-based brain-computer interface.</a> IEEE Trans. Neural Syst. Rehabil. Eng. 14, 234-239 (2006).</li><li>Chae, Y., Jeong, J., Jo, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Toward+brain-actuated+humanoid+robots:+asynchronous+direct+control+using+an+EEG-based+BCI.">Toward brain-actuated humanoid robots: asynchronous direct control using an EEG-based BCI.</a> IEEE T. Robot. 28, 1131-1144 (2012).</li><li>Croft, R. J., Barry, R. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Removal+of+ocular+artifact+from+the+EEG:+a+review.">Removal of ocular artifact from the EEG: a review.</a> Neurophysiol. Clin. 30, 5-19 (2000).</li><li>Hwang, H. J., Hwan Kim, D., Han, C. H., Im, C. H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+new+dual-frequency+stimulation+method+to+increase+the+number+of+visual+stimuli+for+multi-class+SSVEP-based+brain-computer+interface+(BCI).">A new dual-frequency stimulation method to increase the number of visual stimuli for multi-class SSVEP-based brain-computer interface (BCI).</a> Brain Res. 1515, 66-77 (2013).</li><li>Wang, M., Daly, I., Allison, B., Jin, J., Zhang, Y., Chen, L., Wang, X. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+new+hybrid+BCI+paradigm+based+on+P300+and+SSVEP.">A new hybrid BCI paradigm based on P300 and SSVEP.</a> J Neurosci Methods. 244, 16-25 (2015).</li><li>Allison, B. Z., Jin, J., Zhang, Y., Wang, X. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+four-choice+hybrid+P300+SSVEP+BCI+for+improved+accuracy.">A four-choice hybrid P300 SSVEP BCI for improved accuracy.</a> Brain-Computer Interfaces. 1, 17-26 (2014).</li><li>Pan, J. H., Xie, Q., Herman, Y., Wang, F., Di, H., Laureys, S., Yu, R., Li, Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Detecting+awareness+in+patients+with+disorders+of+consciousness+using+a+hybrid+brain-computer+interface.">Detecting awareness in patients with disorders of consciousness using a hybrid brain-computer interface.</a> J. Neural. Eng. 11, 056007 (2014).</li><li>Li, J., Ji, H., Cao, L., Zhang, D., Gu, R., Xia, B., Wu, Q. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Evaluation+and+application+of+a+hybrid+brain+computer+interface+for+real+wheelchair+parallel+control+with+multi-degree+of+freedom.">Evaluation and application of a hybrid brain computer interface for real wheelchair parallel control with multi-degree of freedom.</a> Int J Neural Syst. 24, 1450014-14 (2014).</li><li>Zhao, J., Meng, Q., Li, W., Li, M., Chen, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=SSVEP-based+hierarchical+architecture+for+control+of+a+humanoid+robot+with+mind.">SSVEP-based hierarchical architecture for control of a humanoid robot with mind.</a> Proc. 11th World Congress on Intelligent Control and Automation. , 2401-2406 (2014).</li><li>Zhu, D., Bieger, J., Molina, G. G., Aarts, R. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+survey+of+stimulation+methods+used+in+SSVEP-based+BCIs.">A survey of stimulation methods used in SSVEP-based BCIs.</a> Comput Intell Neurosci. 2010, 1-12 (2010).</li><li>Muller-Putz, G. R., Scherer, R., Brauneis, C., Pfurtscheller, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Steady-state+visual+evoked+potential+(SSVEP)-based+communication:+impact+of+harmonic+frequency+components.">Steady-state visual evoked potential (SSVEP)-based communication: impact of harmonic frequency components.</a> J Neural Eng. 2, 123-130 (2005).</li></ol>

This paper presents an SSVEP-based experimental procedure to establish the brain-robot interaction system with humanoid robots by integrating multiple software programs. Because human intent is perceived by interpreting real-time EEG signals, it is critical to verify the electrode connections and EEG signal qualities before conducting the experiment. If signals acquired from all the electrodes are of poor qualities, it is necessary to check the connection of the ground and reference electrodes first. If there are problem...

Brain-Robot Interaction (BRI), which provides an innovative communication pathway between human and a robotic device via brain signals, is prospective in helping the disabled in their daily lives 1,2. A variety of methods are able to acquire brain signals either invasively or non-invasively, such as electrocorticography (ECoG), electroencephalograph (EEG), functional magnetic resonance imaging (fMRI), etc. The most commonly used non-invasive method for building the BRI system is to acquire EEG signals from electrodes placed on the scalp. This method is inexpensive, easy to use, and provides an acceptable temporal resolution 3. Among a variety of robotic devices, humanoid robots are advanced as they are created to imitate some of the same physical and mental tasks that humans undergo daily. BRI with a humanoid robot will play an important role in assisting the sick and elderly, as well as performing unsanitary or dangerous jobs. But control of a humanoid robot through BRI system is highly challenging, as the humanoid robot with full body movement is developed to perform complex tasks such as personal assistance 4, 5.
Steady-State Visual Evoked Potential (SSVEP) is a type of brain signal evoked by the modulation of visual stimulus at a given frequency 6. It contains sinusoids at the fundamental and harmonic frequencies of the flickering stimulus, and prominently appears throughout the visual cortex in the occipital region of the scalp 7. The reason for choosing the SSVEP signals is that the SSVEP-based BRI system yields relatively high information transfer rate and requires less training 8. Other types of brainwaves, such as event-related potentials (ERPs) 9 or motor-imagery (MI) potentials 10, can also be embedded into this experimental procedure.
Our procedure for brain-robot interaction with humanoid robots is based on Cerebot - a mind-controlled humanoid robot platform - consisting of an EEG data acquisition system and a humanoid robot 11. The EEG system is able to record, pre-process and display bio-potential signals acquired by various types of electrodes. It provides multiple analog I/Os and digital I/Os and is capable of recording up to 128 signal channels simultaneously at a sampling rate of 30 kHz with 16-bits resolution. Its software development kits in C++ and MATLAB are easy for users to design the experimental procedures. The humanoid robot has 25 degrees of freedom and is equipped with multiple sensors, including 2 cameras, 4 microphones, 2 sonar rangefinders, 2 IR emitters and receivers, 1 inertial board, 9 tactile sensors, and 8 pressure sensors. It provides Choregraphe and C++ SDK for creating and editing movements and interactive robot behaviors.
The overall goal of this method is to establish an SSVEP-based experimental procedure by integrating multiple software programs, such as OpenViBE, Choregraph, Central software as well as user developed programs written in C++ and MATLAB, to enable the study of brain-robot interaction with humanoid robots 11. Figure 1 shows the system structure. The dedicated stimulus presentation computer (SPC) displays the User Interface to provide the subject with visual stimuli, instructions and environmental feedbacks. The dedicated data processing computer (DPC) runs the Data Recorder and the Offline Data Analyzer in the offline training process, and runs the Online Signal Processor and the Robot Controller for the online control of the humanoid robot. Compared with other SSVEP-based control systems, our system is more reliable, more flexible, and especially more convenient to be reused and upgraded as it is developed by integrating a number of standardized software packages, such as OpenViBE, Choregraph, Central software, and modules written in C++ and MATLAB.
The following procedure was reviewed and approved by Tianjin medical university general hospital ethics committee, and all subjects gave written consent.

<table border="0" cellpadding="0" cellspacing="0" width="748">
	<tbody>
		<tr height="21">
			<td height="21" style="height:21px;width:284px;">Cerebus EEG Data Acquisition System</td>
			<td style="width:179px;">Blackrock Microsystems</td>
			<td style="width:119px;">4176-9967</td>
			<td style="width:167px;"></td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:284px;">NAO humanoid robot</td>
			<td style="width:179px;">Aldebaran Robotics</td>
			<td style="width:119px;">H25</td>
			<td></td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:284px;">EEG cap</td>
			<td style="width:179px;">Neuroscan</td>
			<td style="width:119px;">8732</td>
			<td></td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:284px;">Ten20 Conductive gel</td>
			<td style="width:179px;">Weaver and company</td>
			<td style="width:119px;">10-20-8</td>
			<td></td>
		</tr>
	</tbody>
</table>

ssvep-based experimental procedure for brain-robot interaction with humanoid robots

Brain-Robot Interaction (BRI), which provides an innovative communication pathway between human and a robotic device via brain signals, is prospective in helping the disabled in their daily lives. The overall goal of our method is to establish an SSVEP-based experimental procedure by integrating multiple software programs, such as OpenViBE, Choregraph, and Central software as well as user developed programs written in C++ and MATLAB, to enable the study of brain-robot interaction with humanoid robots.
This is achieved by first placing EEG electrodes on a human subject to measure the brain responses through an EEG data acquisition system. A user interface is used to elicit SSVEP responses and to display video feedback in the closed-loop control experiments. The second step is to record the EEG signals of first-time subjects, to analyze their SSVEP features offline, and to train the classifier for each subject. Next, the Online Signal Processor and the Robot Controller are configured for the online control of a humanoid robot. As the final step, the subject completes three specific closed-loop control experiments within different environments to evaluate the brain-robot interaction performance.
The advantage of this approach is its reliability and flexibility because it is developed by integrating multiple software programs. The results show that using this approach, the subject is capable of interacting with the humanoid robot via brain signals. This allows the mind-controlled humanoid robot to perform typical tasks that are popular in robotic research and are helpful in assisting the disabled.

The results presented here were obtained from a male subject having corrected-to-normal version. Figure 7 shows the procedure of processing EEG data, including extracting a multichannel data epoch (Figure 7A), spatially filtering the data using CCA coefficients (Figure 7B), and calculating the normalized PSD (Figure 7C).
Figure 8 shows the norma...

Watch this Scientific Journal Video about SSVEP-based Experimental Procedure for Brain-Robot Interaction with Humanoid Robots at JoVE.com

SSVEP-based Experimental Procedure for Brain-Robot Interaction with Humanoid Robots

The overall goal of this method is to establish an SSVEP-based experimental procedure by integrating multiple software programs to enable the study of brain-robot interaction with humanoid robots, which is prospective in assisting the sick and elderly as well as performing unsanitary or dangerous jobs.

Brain-Robot Interaction (BRI), which provides an innovative communication pathway between human and a robotic device via brain signals, is prospective in ...

Research

Education

JoVE Core

JoVE Journal

Electrical Engineering

Engineering

Unit 1

Introduction to Control Systems

SCIENTISTS IN ACTION

Advanced Experimental Methods for Low-temperature Magnetotransport Measurement of Novel Materials

Novel electronic materials are often produced for the first time by synthesis processes that yield bulk crystals (in contrast to single crystal thin film ...

Experimental Procedure for Warm Spinning of Cast Aluminum Components

High performance, cast aluminum automotive wheels are increasingly being incrementally formed via flow forming/metal spinning at elevated temperatures to ...

Experimental Methods for Trapping Ions Using Microfabricated Surface Ion Traps

Ions trapped in a quadrupole Paul trap have been considered one of the strong physical candidates to implement quantum information processing. This is due ...

Experimental Methods for Spin- and Angle-Resolved Photoemission Spectroscopy Combined with Polarization-Variable Laser

The goal of this protocol is to present how to perform spin- and angle-resolved photoemission spectroscopy combined with polarization-variable 7-eV laser ...

Enhanced Electron Injection and Exciton Confinement for Pure Blue Quantum-Dot Light-Emitting Diodes by Introducing Partially Oxidized Aluminum Cathode

Stable and efficient red (R), green (G), and blue (B) light sources based on solution-processed quantum dots (QDs) play important roles in next-generation ...

Experimental Procedure for Laboratory Studies of In Situ Burning : Flammability and Burning Efficiency of Crude Oil

Experimental Procedure for Laboratory Studies of In Situ Burning : Flammability and Burning Efficiency of Crude Oil

A new method for the simultaneous study of the flammability and burning efficiency of fresh and weathered crude oil through two experimental laboratory ...

Theoretical Calculation and Experimental Verification for Dislocation Reduction in Germanium Epitaxial Layers with Semicylindrical Voids on Silicon

Reduction of threading dislocation density (TDD) in epitaxial germanium (Ge) on silicon (Si) has been one of the most important challenges for the ...

Procedure for the Transfer of Polymer Films Onto Porous Substrates with Minimized Defects

The fabrication of devices containing thin film composite membranes necessitates the transfer of these films onto the surfaces of arbitrary support ...

Rapid Manufacturing of Thin Soft Pneumatic Actuators and Robots

This protocol describes a method for rapid manufacturing of soft pneumatic actuators and robots with an ultrathin form factor using a heat press and a ...

Measurement of Chladni Mode Shapes with an Optical Lever Method

Quantitatively determining the Chladni pattern of an elastic plate is of great interest in both physical science and engineering applications. In this ...