The authors would like to express their gratitude to Mr. Hong Hu for his assistance in performing the experiments reported in this paper.&#160;&#160;This work was supported in part by the National Natural Science Foundation of China (No. 61473207).

1. Acquiring EEG Signals
<ol>
	<li>Explain the experimental procedure to the subject and obtain written informed consent to participate in experiments.</li>
	<li>Measure the circumference of the subject&#39;s head using a tape measure and select the EEG cap size that is close to the measurement. The electrodes arrangement is based on the &#34;International 10-20 System&#34; 12.</li>
	<li>Measure the distance between the nasion and inion. Use a skin marker pencil to mark 10% of the distance as a reference for aligning the cap, and mark the midpoint of the distance as the vertex on the subject&#39;s scalp.</li>
	<li>Place the EEG cap on the subject by aligning the 10% mark with the midpoint of the FP1 and FP2 electrodes. Position electrode Cz of the cap on the vertex.</li>
	<li>Make sure that the Fz, Cz, Pz and Oz are on the midline of the head, and that each pair of electrodes located on either side of the midline are in a horizontal line.</li>
	<li>Smear the REF1 and REF2 electrodes with conductive gel. Place the reference electrodes on the left and right mastoids using medical tape. Tighten the chinstrap.</li>
	<li>Place a blunt-tipped syringe in the electrode holders and inject conductive gel into each electrode in the following order: first, the &#34;ground&#34; electrode on the forehead and second, the five electrodes used in the experiment, O2, O1, Oz, Pz, and Cz.</li>
	<li>Seat the subject in a comfortable chair 60 cm in front of a stimulus monitor. Instruct the subject to keep his /her eyes in the same horizontal level with the center of the monitor.</li>
	<li>Connect the electrode wires to the EEG data acquisition system. Configure the sampling rate to 1 kHz.</li>
	<li>Examine the EEG signal quality on the dedicated DPC. If there is a problem with a particular electrode, re-inject gel to adjust the impedance of the channel.</li>
	<li>Use the dedicated SPC to flicker four robot images at frequencies of 4.615, 12, 15 and 20 Hz as visual stimuli on the User Interface as shown in Figure 2.</li>
</ol>
2. Offline Analyzing SSVEP Features
<ol>
	<li>Conduct 32 trials of offline training experiments for each subject and record their brain signals acquired from the EEG system throughout this process. 
	NOTE: this process is only conducted by first-time subjects of the experiment to establish their SSVEP feature vectors and to train the classifier.</li>
	<li>When a trial starts, select one stimulus as the target randomly and display a yellow arrow above it.</li>
	<li>1 sec later, flicker the four visual stimuli at different frequencies on the User Interface for 5 sec.</li>
	<li>Request the subject to focus on the selected stimulus target while keeping his/her body movement to a minimum.</li>
	<li>After each trial, give the subject 3 sec to relax, and then start the next trial.</li>
	<li>When all trials are finished, read the saved data. Extract a 3-sec data epoch between 2 sec and 5 sec after the trigger which selects a stimulus as the target in each trial.</li>
	<li>Calculate the power spectrum density (PSD) of the data epochs using the Offline Data Analyzer in MATLAB scripts. Use the processing algorithm described in details below:
	<ol>
		<li>Calculate the canonical correlation analysis (CCA) coefficients of the multichannel EEG data with a reference data. CCA is a multivariable statistical method used for two sets of data to find their underlying correlation 13. Considering the 3s segment of multichannel EEG data X and reference data Y of the same length, use CCA to find the weight vectors, Wx and Wy, to maximize the correlation coefficient&#160;&#961; between x = XTWx and y = YTWy. Set the reference data to be the periodic signals at certain stimulus frequencies. 
		<img alt="Equation 1" src="/files/ftp_upload/53558/53558eq1.jpg" /> 
		where f1, f2, f3, f4 are the feature frequencies of the four visual stimuli.</li>
		<li>Spatially filter the multichannel EEG data X using the calculated CCA coefficients Wx to obtain an one-dimension filtered data x, which has the most prominent correlation with the linear combination of the reference data.</li>
		<li>Calculate the PSD of the spatially filtered data x using Fast Fourier Transform (FFT). 
		<img alt="Equation 2" src="/files/ftp_upload/53558/53558eq2.jpg" /> 
		where NFFT is the sample size of data x, and FFT(x, NFFT) returns the NFFT-point discrete Fourier transform of x.</li>
		<li>Normalize the PSD with respect to its mean value between 3 and 30 Hz. 
		<img alt="Equation 3" src="/files/ftp_upload/53558/53558eq3.jpg" /> 
		where&#160;<img alt="Equation 3 Denominator" src="/files/ftp_upload/53558/53558eq3denominator.jpg" /> denotes the average of the power spectrum between 3 and 30Hz.</li>
	</ol>
	</li>
	<li>Calibrate the classification parameters for the four stimulus frequencies. First, observe the normalized PSD spectrum that corresponds to its stimulus target, and manually configure the frequency with the strongest PSD amplitude as the feature frequency. Based on the feature frequency, divide the normalized PSD data into two sets: one is acquired when the corresponding stimulus is selected as the target, and the other is acquired when the non-target stimuli are selected. Calculate the mean value of each set respectively. Determine the threshold by the median of the two mean values to classify the target and non-target sets 14.</li>
</ol>
3. Online Processing Brain Signals
<ol>
	<li>Open the Online Signal Processor as shown in Figure 3, which is developed based on the OpenViBE environment and the MATLAB scripts, on the DPC for online processing brain signals.</li>
	<li>Double click the Acquire and Process Signal box, which is marked with a red cycle in Figure 3, to open the OpenViBE setting window. Configure the classification parameters for the subject: set the sampling rate as 1 kHz, set the time length for FFT as 3 sec, and set the feature frequencies according to the results of offline analysis, e.g., 4.667, 24, 15 and 20 Hz.</li>
	<li>Click the Starts button to run the Online Signal Processor, which processes the real-time data using the following algorithm in three steps.</li>
	<li>First, the algorithm acquires EEG data from channels Oz, O1, O2, Pz and CPz of the EEG system every 0.5s, and extracts the data segment of the last 3s for online processing.</li>
	<li>Second, the algorithm processes the 3-sec data segment using the algorithms described in step 2.7, and calculates the real-time PSD for classification.</li>
	<li>Third, the algorithm classifies the brainwave patterns according to the PSD amplitudes at the four feature frequencies. When the amplitude of one feature frequency is above a given threshold, classify the stimulus flickering at the corresponding frequency as the SSVEP target.</li>
</ol>
4. Connecting the Humanoid Robot
<ol>
	<li>Press the Chest button of the humanoid robot and wait for it to start up. 
	NOTE: this process takes approximately 1 min and is completed when the robot says &#34;OGNAK GNOUK&#34; while its Chest button becomes white.</li>
	<li>Establish its Wireless Fidelity (WiFi) connection to the DPC 15.</li>
	<li>Open the Robot Controller programmed in Visual C++ scripts on the DPC. The controller receives the classification results from the Online Signal Processor via Virtual-Reality Peripheral Network (VRPN) interface, translates them into the robot commands, and controls the corresponding behaviors of the humanoid robot via wireless connection.</li>
	<li>Configure the Robot Controller by inputting the IP address of the robot according to its WiFi connection, defining three sets of robot behaviors which can be easily switched to perform the three different tasks, and setting the execution parameters of these behaviors (e.g., walking speed or distance) to fit the task requirement. 
	NOTE: more details of these behaviors are described in steps 5.2.1, 5.3.1, and 5.4.1.</li>
	<li>Click the Build button of the Visual Studio platform to run the Robot Controller.</li>
	<li>Open the Choregraphe program on the SPC. Click &#34;Connect to&#34; button to pop up a Connection widget. Select the robot to be controlled according to its IP address, and double click its icon to connect to it.</li>
	<li>Open the video monitor from the view menu in the Choregraphe menu bar, and move the monitor to the front of the User Interface to display live video feedback from the robot&#39;s camera.</li>
</ol>
5. Conducting Closed-loop Control Experiments
<ol>
	<li>Request the subject to perform three specific closed-loop control tasks within different environments in order to evaluate the brain-robot interaction performance. 
	NOTE: these tasks, which are important in robotic applications because they are helpful in assisting the disabled and elderly in their daily lives, need to activate different robot behaviors.</li>
	<li>Conduct the first closed-loop control task to telepresence control the humanoid robot via brain signals to walk through obstacles and push a light switch to turn on the light, as shown in Figure 4.
	<ol>
		<li>To accomplish the first task, control the robot behaviors which are encoded by the four SSVEP stimuli as: stepping forward with a fixed distance of 0.15 m, turning left with a fixed radian of 0.3 rad, turning right with a fixed radian of 0.3 rad, and pushing the switch using its right hand. 
		NOTE: the configuration process is described in step 4.4.</li>
		<li>Inform the subject of the task objective and the four behaviors to be controlled.</li>
		<li>Run the Robot Controller as in steps 4.5-4.7, and start the experiment. Give the subject the freedom to perceive the environment and make decisions based on live video feedback, and to activate the robot behaviors by staring at the corresponding stimuli. 
		NOTE: no instruction or help should be given to the subject unless in emergency situations, e.g., the robot is about to collide with an obstacle.</li>
	</ol>
	</li>
	<li>Conduct the second task to telepresence control the humanoid robot via brain signals to walk toward the staircase following the exit sign, as shown in Figure 5. When confronting passersby, ask the subject to control the robot to say &#34;excuse me&#34; and wait for the passersby to give way.
	<ol>
		<li>Control the robot behaviors encoded by the four SSVEP stimuli as: walking forward continuously at a speed of 0.05 m/sec, turning left continuously at a speed of 0.1 rad/sec, turning right continuously at a speed of 0.1 rad/sec, and stopping all the walking behaviors.</li>
		<li>Switch the Robot Controller to the posture-dependent control function 16. 
		NOTE: when the robot is walking, this function uses the fourth stimulus as the command of stopping the walking behaviors; when the robot is not walking, it uses the fourth stimulus to control the robot to say &#34;excuse me&#34;. The function is implemented in C++ scripts by detecting the robot&#39;s walking status.</li>
		<li>Inform the subject of the task objective and the five behaviors to be controlled.</li>
		<li>Run the Robot Controller and start the experiment, as described in step 5.2.3.</li>
	</ol>
	</li>
	<li>Conduct the third task to telepresence control the humanoid robot to walk toward a balloon target, to pick it up, and to deliver it to the subject&#39;s hand, as shown in Figure 6.
	<ol>
		<li>Control the robot behaviors encoded by the four SSVEP stimuli as: stepping forward with a fixed distance of 0.15 m, turning left with a fixed radian of 0.3 rad, turning right with a fixed radian of 0.3 rad, and picking up objects.</li>
		<li>Activate the posture-dependent control function in the Robot Controller to reuse the fourth stimulus as the command of putting down the object when it has already been picked up.</li>
		<li>Inform the subject of the task objective and the five behaviors to be controlled.</li>
		<li>Run the Robot Controller and start the experiment, as described in step 5.2.3.</li>
	</ol>
	</li>
</ol>

<ol>
	<li>McFarland, D. J., Wolpaw, J. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Brain-Computer+Interface+Operation+of+Robotic+and+Prosthetic+Devices.">Brain-Computer Interface Operation of Robotic and Prosthetic Devices.</a> Computer. 41, 52-56 (2008).</li><li>Lebedev, M. A., Nicolelis, M. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Brain-machine+interfaces:+Past,+present+and+future.">Brain-machine interfaces: Past, present and future.</a> Trends Neruosci. 29 (9), 536-546 (2006).</li><li>Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., Vaughan, T. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Brain-computer+interfaces+for+communication+and+control.">Brain-computer interfaces for communication and control.</a> Clin. Neurophysiol. 113, 767-791 (2002).</li><li>Bell, C. J., Shenoy, P., Chalodhorn, R., Rao, R. P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Control+of+a+humanoid+robot+by+a+noninvasive+brain-computer+interface+in+humans.">Control of a humanoid robot by a noninvasive brain-computer interface in humans.</a> J. Neural. Eng. 5, 214-220 (2008).</li><li>Li, W., Li, M., Zhao, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Control+of+humanoid+robot+via+motion-onset+visual+evoked+potentials.">Control of humanoid robot via motion-onset visual evoked potentials.</a> Front. Syst. Neurosci. 8, 247 (2014).</li><li>Regan, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Some+characteristics+of+average+steady-state+and+transient+responses+evoked+by+modulated+light.">Some characteristics of average steady-state and transient responses evoked by modulated light.</a> Electroencephalogr. Clin. Neurophysiol. 20, 238-248 (1966).</li><li>Vialatte, F. B., Maurice, M., Dauwels, J., Cichocki, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Steady-state+visually+evoked+potentials:+focus+on+essential+paradigms+and+future+perspectives.">Steady-state visually evoked potentials: focus on essential paradigms and future perspectives.</a> Prog. Neurobiol. 90, 418-438 (2010).</li><li>Bin, G., Gao, X., Wang, Y., Li, Y., Hong, B., Gao, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+high-speed+BCI+based+on+code+modulation+VEP.">A high-speed BCI based on code modulation VEP.</a> J. Neural. Eng. 8, 025015 (2011).</li><li>Sutton, S., Braren, M., Zubin, J., John, E. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Evoked-potential+correlates+of+stimulus+uncertainty.">Evoked-potential correlates of stimulus uncertainty.</a> Science. 150, 1187-1188 (1965).</li><li>Pfurtscheller, G., Lopes da Silva, H. F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Event-related+EEG/MEG+synchronization+and+desynchronization:+basic+principles.">Event-related EEG/MEG synchronization and desynchronization: basic principles.</a> Clin. Neurophysiol. 110, 1842-1857 (1999).</li><li>Zhao, J., Meng, Q., Li, W., Li, M., Sun, F., Chen, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=OpenViBE-based+brainwave+control+system+for+Cerebot.">OpenViBE-based brainwave control system for Cerebot.</a> Proc. IEEE International Conference on Robotics and Biomimetics. , 1169-1174 (2013).</li><li>Homan, R. W., Herman, J., Purdy, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Cerebral+location+of+international+10-20+system+electrode+placement.">Cerebral location of international 10-20 system electrode placement.</a> Electroencephalogr. Clin. Neurophysiol. 66, 376-382 (1987).</li><li>Bin, G., Gao, X., Yan, Z., Hong, B., Gao, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=An+online+multi-channel+SSVEP-based+brain-computer+interface+using+a+canonical+correlation+analysis+method.">An online multi-channel SSVEP-based brain-computer interface using a canonical correlation analysis method.</a> J. Neural. Eng. 6, 046002 (2009).</li><li>Wang, Y., Wang, R., Gao, X., Hong, B., Gao, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+practical+VEP-based+brain-computer+interface.">A practical VEP-based brain-computer interface.</a> IEEE Trans. Neural Syst. Rehabil. Eng. 14, 234-239 (2006).</li><li>Chae, Y., Jeong, J., Jo, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Toward+brain-actuated+humanoid+robots:+asynchronous+direct+control+using+an+EEG-based+BCI.">Toward brain-actuated humanoid robots: asynchronous direct control using an EEG-based BCI.</a> IEEE T. Robot. 28, 1131-1144 (2012).</li><li>Croft, R. J., Barry, R. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Removal+of+ocular+artifact+from+the+EEG:+a+review.">Removal of ocular artifact from the EEG: a review.</a> Neurophysiol. Clin. 30, 5-19 (2000).</li><li>Hwang, H. J., Hwan Kim, D., Han, C. H., Im, C. H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+new+dual-frequency+stimulation+method+to+increase+the+number+of+visual+stimuli+for+multi-class+SSVEP-based+brain-computer+interface+(BCI).">A new dual-frequency stimulation method to increase the number of visual stimuli for multi-class SSVEP-based brain-computer interface (BCI).</a> Brain Res. 1515, 66-77 (2013).</li><li>Wang, M., Daly, I., Allison, B., Jin, J., Zhang, Y., Chen, L., Wang, X. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+new+hybrid+BCI+paradigm+based+on+P300+and+SSVEP.">A new hybrid BCI paradigm based on P300 and SSVEP.</a> J Neurosci Methods. 244, 16-25 (2015).</li><li>Allison, B. Z., Jin, J., Zhang, Y., Wang, X. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+four-choice+hybrid+P300+SSVEP+BCI+for+improved+accuracy.">A four-choice hybrid P300 SSVEP BCI for improved accuracy.</a> Brain-Computer Interfaces. 1, 17-26 (2014).</li><li>Pan, J. H., Xie, Q., Herman, Y., Wang, F., Di, H., Laureys, S., Yu, R., Li, Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Detecting+awareness+in+patients+with+disorders+of+consciousness+using+a+hybrid+brain-computer+interface.">Detecting awareness in patients with disorders of consciousness using a hybrid brain-computer interface.</a> J. Neural. Eng. 11, 056007 (2014).</li><li>Li, J., Ji, H., Cao, L., Zhang, D., Gu, R., Xia, B., Wu, Q. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Evaluation+and+application+of+a+hybrid+brain+computer+interface+for+real+wheelchair+parallel+control+with+multi-degree+of+freedom.">Evaluation and application of a hybrid brain computer interface for real wheelchair parallel control with multi-degree of freedom.</a> Int J Neural Syst. 24, 1450014-14 (2014).</li><li>Zhao, J., Meng, Q., Li, W., Li, M., Chen, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=SSVEP-based+hierarchical+architecture+for+control+of+a+humanoid+robot+with+mind.">SSVEP-based hierarchical architecture for control of a humanoid robot with mind.</a> Proc. 11th World Congress on Intelligent Control and Automation. , 2401-2406 (2014).</li><li>Zhu, D., Bieger, J., Molina, G. G., Aarts, R. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+survey+of+stimulation+methods+used+in+SSVEP-based+BCIs.">A survey of stimulation methods used in SSVEP-based BCIs.</a> Comput Intell Neurosci. 2010, 1-12 (2010).</li><li>Muller-Putz, G. R., Scherer, R., Brauneis, C., Pfurtscheller, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Steady-state+visual+evoked+potential+(SSVEP)-based+communication:+impact+of+harmonic+frequency+components.">Steady-state visual evoked potential (SSVEP)-based communication: impact of harmonic frequency components.</a> J Neural Eng. 2, 123-130 (2005).</li></ol>

The authors have nothing to disclose.

This paper presents an SSVEP-based experimental procedure to establish the brain-robot interaction system with humanoid robots by integrating multiple software programs. Because human intent is perceived by interpreting real-time EEG signals, it is critical to verify the electrode connections and EEG signal qualities before conducting the experiment. If signals acquired from all the electrodes are of poor qualities, it is necessary to check the connection of the ground and reference electrodes first. If there are problem...

Brain-Robot Interaction (BRI), which provides an innovative communication pathway between human and a robotic device via brain signals, is prospective in helping the disabled in their daily lives 1,2. A variety of methods are able to acquire brain signals either invasively or non-invasively, such as electrocorticography (ECoG), electroencephalograph (EEG), functional magnetic resonance imaging (fMRI), etc. The most commonly used non-invasive method for building the BRI system is to acquire EEG signals from electrodes placed on the scalp. This method is inexpensive, easy to use, and provides an acceptable temporal resolution 3. Among a variety of robotic devices, humanoid robots are advanced as they are created to imitate some of the same physical and mental tasks that humans undergo daily. BRI with a humanoid robot will play an important role in assisting the sick and elderly, as well as performing unsanitary or dangerous jobs. But control of a humanoid robot through BRI system is highly challenging, as the humanoid robot with full body movement is developed to perform complex tasks such as personal assistance 4, 5.
Steady-State Visual Evoked Potential (SSVEP) is a type of brain signal evoked by the modulation of visual stimulus at a given frequency 6. It contains sinusoids at the fundamental and harmonic frequencies of the flickering stimulus, and prominently appears throughout the visual cortex in the occipital region of the scalp 7. The reason for choosing the SSVEP signals is that the SSVEP-based BRI system yields relatively high information transfer rate and requires less training 8. Other types of brainwaves, such as event-related potentials (ERPs) 9 or motor-imagery (MI) potentials 10, can also be embedded into this experimental procedure.
Our procedure for brain-robot interaction with humanoid robots is based on Cerebot - a mind-controlled humanoid robot platform - consisting of an EEG data acquisition system and a humanoid robot 11. The EEG system is able to record, pre-process and display bio-potential signals acquired by various types of electrodes. It provides multiple analog I/Os and digital I/Os and is capable of recording up to 128 signal channels simultaneously at a sampling rate of 30 kHz with 16-bits resolution. Its software development kits in C++ and MATLAB are easy for users to design the experimental procedures. The humanoid robot has 25 degrees of freedom and is equipped with multiple sensors, including 2 cameras, 4 microphones, 2 sonar rangefinders, 2 IR emitters and receivers, 1 inertial board, 9 tactile sensors, and 8 pressure sensors. It provides Choregraphe and C++ SDK for creating and editing movements and interactive robot behaviors.
The overall goal of this method is to establish an SSVEP-based experimental procedure by integrating multiple software programs, such as OpenViBE, Choregraph, Central software as well as user developed programs written in C++ and MATLAB, to enable the study of brain-robot interaction with humanoid robots 11. Figure 1 shows the system structure. The dedicated stimulus presentation computer (SPC) displays the User Interface to provide the subject with visual stimuli, instructions and environmental feedbacks. The dedicated data processing computer (DPC) runs the Data Recorder and the Offline Data Analyzer in the offline training process, and runs the Online Signal Processor and the Robot Controller for the online control of the humanoid robot. Compared with other SSVEP-based control systems, our system is more reliable, more flexible, and especially more convenient to be reused and upgraded as it is developed by integrating a number of standardized software packages, such as OpenViBE, Choregraph, Central software, and modules written in C++ and MATLAB.
The following procedure was reviewed and approved by Tianjin medical university general hospital ethics committee, and all subjects gave written consent.

<table><tbody><tr><td>Cerebus EEG Data Acquisition System</td><td>Blackrock Microsystems</td><td>4176-9967</td><td></td></tr><tr><td>NAO humanoid robot</td><td>Aldebaran Robotics</td><td>H25</td><td></td></tr><tr><td>EEG cap</td><td>Neuroscan</td><td>8732</td><td></td></tr><tr><td>Ten20 Conductive gel</td><td>Weaver and company</td><td>10-20-8</td><td></td></tr></tbody></table>

ssvep-based experimental procedure for brain-robot interaction with humanoid robots

Brain-Robot Interaction (BRI), which provides an innovative communication pathway between human and a robotic device via brain signals, is prospective in helping the disabled in their daily lives. The overall goal of our method is to establish an SSVEP-based experimental procedure by integrating multiple software programs, such as OpenViBE, Choregraph, and Central software as well as user developed programs written in C++ and MATLAB, to enable the study of brain-robot interaction with humanoid robots.
This is achieved by first placing EEG electrodes on a human subject to measure the brain responses through an EEG data acquisition system. A user interface is used to elicit SSVEP responses and to display video feedback in the closed-loop control experiments. The second step is to record the EEG signals of first-time subjects, to analyze their SSVEP features offline, and to train the classifier for each subject. Next, the Online Signal Processor and the Robot Controller are configured for the online control of a humanoid robot. As the final step, the subject completes three specific closed-loop control experiments within different environments to evaluate the brain-robot interaction performance.
The advantage of this approach is its reliability and flexibility because it is developed by integrating multiple software programs. The results show that using this approach, the subject is capable of interacting with the humanoid robot via brain signals. This allows the mind-controlled humanoid robot to perform typical tasks that are popular in robotic research and are helpful in assisting the disabled.

The results presented here were obtained from a male subject having corrected-to-normal version. Figure 7 shows the procedure of processing EEG data, including extracting a multichannel data epoch (Figure 7A), spatially filtering the data using CCA coefficients (Figure 7B), and calculating the normalized PSD (Figure 7C).
Figure 8 shows the norma...

Watch this Scientific Journal Video about SSVEP-based Experimental Procedure for Brain-Robot Interaction with Humanoid Robots at JoVE.com

SSVEP-based Experimental Procedure for Brain-Robot Interaction with Humanoid Robots

The overall goal of this method is to establish an SSVEP-based experimental procedure by integrating multiple software programs to enable the study of brain-robot interaction with humanoid robots, which is prospective in assisting the sick and elderly as well as performing unsanitary or dangerous jobs.

Brain-Robot Interaction (BRI), which provides an innovative communication pathway between human and a robotic device via brain signals, is prospective in ...

ssvep-based-experimental-procedure-for-brain-robot-interaction-with

Research

Education

JoVE Core

JoVE Journal

Electrical Engineering

Engineering

Unit 1

Introduction to Control Systems

SCIENTISTS IN ACTION

13.1K Views.  Tianjin University. The overall goal of this method is to establish an SSVEP-based experimental procedure by integrating multiple software programs to enable the study of brain-robot interaction with humanoid robots, which is prospective in assisting the sick and elderly as well as performing unsanitary or dangerous jobs.

Video: SSVEP-based Experimental Procedure for Brain-Robot Interaction with Humanoid Robots

An Experimental Platform to Study the Closed-loop Performance of Brain-machine Interfaces

The non-stationary nature and variability of neuronal signals is a fundamental problem in brain-machine interfacing. We developed a brain-machine interface to assess the robustness of different control-laws applied to a closed-loop image stabilization task. Taking advantage of the well-characterized fly visuomotor pathway we record the electrical activity from an identified, motion-sensitive neuron, H1, to control the yaw rotation of a two-wheeled robot. The robot is equipped with 2 high-speed video cameras providing visual motion input to a fly placed in front of 2 CRT computer monitors. The activity of the H1 neuron indicates the direction and relative speed of the robot's rotation. The neural activity is filtered and fed back into the steering system of the robot by means of proportional and proportional/adaptive control. Our goal is to test and optimize the performance of various control laws under closed-loop conditions for a broader application also in other brain machine interfaces.

We use a closed-loop fly-machine interface to investigate general principles in neuronal control.

The non-stationary nature and variability of neuronal signals is a fundamental problem in brain-machine interfacing. We developed a brain-machine ...

Neuroscience

Haptic/Graphic Rehabilitation: Integrating a Robot into a Virtual Environment Library and Applying it to Stroke Therapy

Recent research that tests interactive devices for prolonged therapy practice has revealed new prospects for robotics combined with graphical and other forms of biofeedback. Previous human-robot interactive systems have required different software commands to be implemented for each robot leading to unnecessary developmental overhead time each time a new system becomes available. For example, when a haptic/graphic virtual reality environment has been coded for one specific robot to provide haptic feedback, that specific robot would not be able to be traded for another robot without recoding the program. However, recent efforts in the open source community have proposed a wrapper class approach that can elicit nearly identical responses regardless of the robot used. The result can lead researchers across the globe to perform similar experiments using shared code. Therefore modular "switching out"of one robot for another would not affect development time. In this paper, we outline the successful creation and implementation of a wrapper class for one robot into the open-source H3DAPI, which integrates the software commands most commonly used by all robots.

Recently, a vast amount of prospects have come available for human-robot interactive systems. In this paper we outline the integration of a new robotic device with open source software that can rapidly make possible a library of interactive functionality. We then outline a clinical application for a neurorehabilitation application.

Recent research that tests interactive devices for prolonged therapy practice has revealed new prospects for robotics combined with graphical and other ...

Bioengineering

Adaptation of a Haptic Robot in a 3T fMRI

Functional magnetic resonance imaging (fMRI) provides excellent functional brain imaging via the BOLD signal 1 with advantages including non-ionizing radiation, millimeter spatial accuracy of anatomical and functional data 2, and nearly real-time analyses 3. Haptic robots provide precise measurement and control of position and force of a cursor in a reasonably confined space. Here we combine these two technologies to allow precision experiments involving motor control with haptic/tactile environment interaction such as reaching or grasping. The basic idea is to attach an 8 foot end effecter supported in the center to the robot 4 allowing the subject to use the robot, but shielding it and keeping it out of the most extreme part of the magnetic field from the fMRI machine (Figure 1).
The Phantom Premium 3.0, 6DoF, high-force robot (SensAble Technologies, Inc.) is an excellent choice for providing force-feedback in virtual reality experiments 5, 6, but it is inherently non-MR safe, introduces significant noise to the sensitive fMRI equipment, and its electric motors may be affected by the fMRI's strongly varying magnetic field. We have constructed a table and shielding system that allows the robot to be safely introduced into the fMRI environment and limits both the degradation of the fMRI signal by the electrically noisy motors and the degradation of the electric motor performance by the strongly varying magnetic field of the fMRI. With the shield, the signal to noise ratio (SNR: mean signal/noise standard deviation) of the fMRI goes from a baseline of &tilde;380 to &tilde;330, and &tilde;250 without the shielding. The remaining noise appears to be uncorrelated and does not add artifacts to the fMRI of a test sphere (Figure 2). The long, stiff handle allows placement of the robot out of range of the most strongly varying parts of the magnetic field so there is no significant effect of the fMRI on the robot. The effect of the handle on the robot's kinematics is minimal since it is lightweight (&tilde;2.6 lbs) but extremely stiff 3/4" graphite and well balanced on the 3DoF joint in the middle. The end result is an fMRI compatible, haptic system with about 1 cubic foot of working space, and, when combined with virtual reality, it allows for a new set of experiments to be performed in the fMRI environment including naturalistic reaching, passive displacement of the limb and haptic perception, adaptation learning in varying force fields, or texture identification 5, 6.

The adaptation and use of a haptic robot in a 3T fMRI is described.

Functional magnetic resonance imaging (fMRI) provides excellent functional brain imaging via the BOLD signal 1 with advantages including non-ionizing ...

Functional Near Infrared Spectroscopy of the Sensory and Motor Brain Regions with Simultaneous Kinematic and EMG Monitoring During Motor Tasks

There are several advantages that functional near-infrared spectroscopy (fNIRS) presents in the study of the neural control of human movement. It is relatively flexible with respect to participant positioning and allows for some head movements during tasks. Additionally, it is inexpensive, light weight, and portable, with very few contraindications to its use. This presents a unique opportunity to study functional brain activity during motor tasks in individuals who are typically developing, as well as those with movement disorders, such as cerebral palsy. An additional consideration when studying movement disorders, however, is the quality of actual movements performed and the potential for additional, unintended movements. Therefore, concurrent monitoring of both blood flow changes in the brain and actual movements of the body during testing is required for appropriate interpretation of fNIRS results. Here, we show a protocol for the combination of fNIRS with muscle and kinematic monitoring during motor tasks. We explore gait, a unilateral multi-joint movement (cycling), and two unilateral single-joint movements (isolated ankle dorsiflexion, and isolated hand squeezing). The techniques presented can be useful in studying both typical and atypical motor control, and can be modified to investigate a broad range of tasks and scientific questions.

Monitoring brain activity during upright motor tasks is of great value when investigating the neural source of movement disorders. Here, we demonstrate a protocol that combines functional near infrared spectroscopy with continuous monitoring of muscle and kinematic activity during 4 types of motor tasks.

There are several advantages that functional near-infrared spectroscopy (fNIRS) presents in the study of the neural control of human movement. It is ...

Behavior

Multifunctional Setup for Studying Human Motor Control Using Transcranial Magnetic Stimulation, Electromyography, Motion Capture, and Virtual Reality

The study of neuromuscular control of movement in humans is accomplished with numerous technologies. Non-invasive methods for investigating neuromuscular function include transcranial magnetic stimulation, electromyography, and three-dimensional motion capture. The advent of readily available and cost-effective virtual reality solutions has expanded the capabilities of researchers in recreating &#8220;real-world&#8221; environments and movements in a laboratory setting. Naturalistic movement analysis will not only garner a greater understanding of motor control in healthy individuals, but also permit the design of experiments and rehabilitation strategies that target specific motor impairments (e.g. stroke). The combined use of these tools will lead to increasingly deeper understanding of neural mechanisms of motor control. A key requirement when combining these data acquisition systems is fine temporal correspondence between the various data streams. This protocol describes a multifunctional system&#8217;s overall connectivity, intersystem signaling, and the temporal synchronization of recorded data. Synchronization of the component systems is primarily accomplished through the use of a customizable circuit, readily made with off the shelf components and minimal electronics assembly skills.

Transcranial magnetic stimulation, electromyography, and 3D motion capture are commonly used non-invasive techniques for investigating neuromuscular function in humans. In this paper, we describe a protocol that synchronously samples data generated by all three of these tools along with the unique addition of virtual reality stimulus presentation and feedback.

The study of neuromuscular control of movement in humans is accomplished with numerous technologies. Non-invasive methods for investigating neuromuscular ...

Designing and Implementing Nervous System Simulations on LEGO Robots

We present a method to use the commercially available LEGO Mindstorms NXT robotics platform to test systems level neuroscience hypotheses. The first step of the method is to develop a nervous system simulation of specific reflexive behaviors of an appropriate model organism; here we use the American Lobster. Exteroceptive reflexes mediated by decussating (crossing) neural connections can explain an animal's taxis towards or away from a stimulus as described by Braitenberg and are particularly well suited for investigation using the NXT platform.1 The nervous system simulation is programmed using LabVIEW software on the LEGO Mindstorms platform. Once the nervous system is tuned properly, behavioral experiments are run on the robot and on the animal under identical environmental conditions. By controlling the sensory milieu experienced by the specimens, differences in behavioral outputs can be observed. These differences may point to specific deficiencies in the nervous system model and serve to inform the iteration of the model for the particular behavior under study. This method allows for the experimental manipulation of electronic nervous systems and serves as a way to explore neuroscience hypotheses specifically regarding the neurophysiological basis of simple innate reflexive behaviors. The LEGO Mindstorms NXT kit provides an affordable and efficient platform on which to test preliminary biomimetic robot control schemes. The approach is also well suited for the high school classroom to serve as the foundation for a hands-on inquiry-based biorobotics curriculum.

An approach to neural network modeling on the LEGO Mindstorms robotics platform is presented. The method provides a simulation tool for invertebrate neuroscience research in both the research lab and the classroom. This technique enables the investigation of biomimetic robot control principles.

We present a method to use the commercially available LEGO Mindstorms NXT robotics platform to test systems level neuroscience hypotheses. The first step ...

Operative Technique and Nuances for the Stereoelectroencephalographic (SEEG) Methodology Utilizing a Robotic Stereotactic Guidance System

The SEEG methodology has gained favor in North America over the last decade as a means of localizing the epileptogenic zone (EZ) prior to epilepsy surgery. Recently, the application of a robotic stereotactic guidance system for implantation of SEEG electrodes has become more popular in many epilepsy centers. The technique for the use of the robot requires extreme precision in the pre-surgical planning phase and then the technique is streamlined during the operative portion of the methodology, as the robot and surgeon work in concert to implant the electrodes. Herein is detailed precise operative methodology of using the robot to guide implantation of SEEG electrodes. A major limitation of the procedure, namely its heavy reliance on the ability to register the patient to a preoperative volumetric magnetic resonance image (MRI), is also discussed. Overall, this procedure has been shown to have a low morbidity rate and an extremely low mortality rate. The use of a robotic stereotactic guidance system for the implantation of SEEG electrodes is an efficient, fast, safe, and accurate alternative to conventional manual implantation strategies.

Operative Technique and Nuances for the Stereoelectroencephalographic SEEG Methodology Utilizing a Robotic Stereotactic Guidance System

The SEEG methodology is simplified and made faster with a stereotactic robot. Careful attention must be paid to the registration of the preoperative volumetric MRI to the patient prior to use of the robot in the OR. The robot streamlines the procedure, leading to decreased operative times and accurate implantations.

The SEEG methodology has gained favor in North America over the last decade as a means of localizing the epileptogenic zone (EZ) prior to epilepsy ...

Closed-loop Neuro-robotic Experiments to Test Computational Properties of Neuronal Networks

Information coding in the Central Nervous System (CNS) remains unexplored. There is mounting evidence that, even at a very low level, the representation of a given stimulus might be dependent on context and history. If this is actually the case, bi-directional interactions between the brain (or if need be a reduced model of it) and sensory-motor system can shed a light on how encoding and decoding of information is performed. Here an experimental system is introduced and described in which the activity of a neuronal element (i.e., a network of neurons extracted from embryonic mammalian hippocampi) is given context and used to control the movement of an artificial agent, while environmental information is fed back to the culture as a sequence of electrical stimuli. This architecture allows a quick selection of diverse encoding, decoding, and learning algorithms to test different hypotheses on the computational properties of neuronal networks.

In this paper, an experimental framework to perform closed-loop experiments is presented, in which information processing (i.e., coding and decoding) and learning of neuronal assemblies are studied during the continuous interaction with a robotic body.

Information coding in the Central Nervous System (CNS) remains unexplored. There is mounting evidence that, even at a very low level, the representation ...

Motor Imagery Using VR Digital Avatars in a Brain-Computer Interface System

Motor Imagery Performance Through Embodied Digital Twins in a Virtual Reality-Enabled Brain-Computer Interface Environment

This study introduces an innovative framework for neurological rehabilitation by integrating brain-computer interfaces (BCI) and virtual reality (VR) technologies with the customization of three-dimensional (3D) avatars. Traditional approaches to rehabilitation often fail to fully engage patients, primarily due to their inability to provide a deeply immersive and interactive experience. This research endeavors to fill this gap by utilizing motor imagery (MI) techniques, where participants visualize physical movements without actual execution. This method capitalizes on the brain's neural mechanisms, activating areas involved in movement execution when imagining movements, thereby facilitating the recovery process. The integration of VR's immersive capabilities with the precision of electroencephalography (EEG) to capture and interpret brain activity associated with imagined movements forms the core of this system. Digital Twins in the form of personalized 3D avatars are employed to significantly enhance the sense of immersion within the virtual environment. This heightened sense of embodiment is crucial for effective rehabilitation, aiming to bolster the connection between the patient and their virtual counterpart. By doing so, the system not only aims to improve motor imagery performance but also seeks to provide a more engaging and efficacious rehabilitation experience. Through the real-time application of BCI, the system allows for the direct translation of imagined movements into virtual actions performed by the 3D avatar, offering immediate feedback to the user. This feedback loop is essential for reinforcing the neural pathways involved in motor control and recovery. The ultimate goal of the developed system is to significantly enhance the effectiveness of motor imagery exercises by making them more interactive and responsive to the user's cognitive processes, thereby paving a new path in the field of neurological rehabilitation.

Author Spotlight: Enhancing Neurorehabilitation Through EEG, Motor Imagery, and Virtual Reality

Motor imagery in a virtual reality environment has wide applications in brain-computer interface systems. This manuscript outlines the use of personalized digital avatars that resemble the participants performing movements imagined by the participant in a virtual reality environment to enhance immersion and a sense of body ownership.

This study introduces an innovative framework for neurological rehabilitation by integrating brain-computer interfaces (BCI) and virtual reality (VR) ...

Evaluating Brain Activation by fNIRS During a Lower Limb Robotic Therapy Session

An Experiment Using Functional Near-Infrared Spectroscopy and Robot-Assisted Multi-Joint Pointing Movements of the Lower Limb

Stroke affects approximately 17 million individuals worldwide each year and is a leading cause of long-term disability. Robotic therapy has shown promise in helping stroke patients regain lost motor functions. One potential avenue for increasing the understanding of how motor recovery occurs is to study brain activation during the movements that are targeted by therapy in healthy individuals. Functional Near-Infrared Spectroscopy (fNIRS) has emerged as a promising neuroimaging technique for examining neural underpinnings of motor function. This study aimed to investigate fNIRS neural correlates of complex lower limb movements in healthy subjects. Participants were asked to perform cycles of rest and movement for 6 min using a robotic device for motor rehabilitation. The task required coordinated knee and ankle joint movements to point to targets displayed on a computer screen. Two experimental conditions with different levels of movement assistance provided by the robot were explored. The results showed that the fNIRS protocol effectively detected brain regions associated with motor control during the task. Notably, all subjects exhibited greater activation in the contralateral premotor area during the no-assistance condition compared to the assisted condition. In conclusion, fNIRS appears to be a valuable approach for detecting changes in oxyhemoglobin concentration associated with multi-joint pointing movements of the lower limb. This research might contribute to the understanding of stroke motor recovery mechanisms and might pave the way for improved rehabilitation treatments for stroke patients. However, further research is needed to fully elucidate the potential of fNIRS in studying motor function and its applications in clinical settings.

Author Spotlight: Assessing Brain Activity in Robotic-Assisted Lower Limb Rehabilitation Using fNIRS

It is estimated that 1 in 6 individuals worldwide will have a stroke in their lifetime, causing long-term disability, whose rehabilitation mechanisms are still poorly understood. This study proposes a protocol to evaluate brain activation by functional near-infrared spectroscopy (fNIRS) during a lower limb robotic therapy session.

Stroke affects approximately 17 million individuals worldwide each year and is a leading cause of long-term disability. Robotic therapy has shown promise ...