The authors acknowledge the financial support of the Social Sciences and Humanities Research Council of Canada (SSHERC), Natural Sciences and Engineering Research Council of Canada (NSERC), Fonds de Recherche Nature et Technologies du Qu&#233;bec (FQRNT) and Fonds de Recherche sur la Soci&#233;t&#233; et Culture du Qu&#233;bec (FQRSC).

This protocol received an ethical certificate from the Comit&#233; institutionnel de la recherche avec des &#234;tres humains (CIER) de l&#8217;Universit&#233; du Qu&#233;bec &#224; Montr&#233;al (UQAM) that was endorsed by HEC-Montreal for the Tech3Lab research facility. The protocol describes each of the specific steps that are performed in our lab environment and equipment. Although precise software paths are provided to clarify the methodology, this technique is transferable and can be replicated with other proprietary eye-tracking, automatic facial emotion recognition, electrodermal activity and electroencephalography equipment and software.
1. Setup of the Lab Environment
<ol>
	<li>Turn on the eye-tracker, the EEG amplifier, the four recording computers and the speakers.</li>
	<li>Prepare the setup of the recording equipment:
	<ol>
		<li>Prepare the EEG setup with required material according to the manufacturer&#8217;s recommended procedures. Prepare the EEG software for the upcoming participant. Start the eye-tracking software and create a new participant profile in the software. Start the video recording software and the cameras.</li>
		<li>Start the synchronization software with the specific subroutine created for the project with markers at 60&#160;sec. Start the physiological measurement software (to record electrodermal activity) and open the specific layout created for the project. Adjust the participant&#8217;s chair to the highest level.</li>
	</ol>
	</li>
</ol>
2. Participant Preparation
<ol>
	<li>Ask the participant to read and sign the ethical consent form.</li>
	<li>Carry out skull measurements for EEG:
	<ol>
		<li>Find the Cz location on the participant&#8217;s head (according to 10 - 20 reference systems). Immerse the EEG net in the saline solution (potassium chloride) (see step 1.2.1) and start a timer (10 min) in accordance with the manufacturer&#8217;s standards.</li>
	</ol>
	</li>
	<li>Read the objective of the study and the steps in the experiment to the participant, &#8220;The objective of this study is to observe your brain activity while you answer physics problems. First we will install the sensors, then you will be asked to solve 10 Newtonian physics problems on the computer. We will ask you to take a 45-sec&#160;break after each problem with your eyes closed. After each problem, you will be asked to rate your assessment of the problem.&#8221;</li>
	<li>Tell the subject that the total duration of the experiment will be 90 min.</li>
	<li>Install the physiological sensors, according to the manufacturer&#8217;s recommendations: two gelified sensors on the top of the left hand.</li>
	<li>Install the EEG cap, according to the manufacturer&#8217;s recommendations and perform an impedance check with a threshold at 40 k&#937; (according to the manufacturer&#8217;s specifications).</li>
</ol>
3. Data Collection
<ol>
	<li>Make sure that all the recording software is ready to be started in synchrony:
	<ol>
		<li>Physiology (EDA data): Click the &#8220;start&#8221; button.</li>
		<li>Video recording: Click the &#8220;open&#8221; button.</li>
		<li>Eye-tracking: Click the &#8220;on hold&#8221; button.</li>
		<li>EEG: Click the &#8220;record&#8221; button.</li>
		<li>Synchronization software: Click the &#8220;green circle&#8221; button.</li>
	</ol>
	</li>
	<li>Eye gaze calibration:
	<ol>
		<li>Perform a five-point on-screen calibration and observe the participant while he/she follows the red dots (click &#8220;Tools/Settings/Calibration &#8230;&#8221;). Repeat this procedure until sufficient accuracy is achieved, according to the manufacturer&#8217;s standards.</li>
	</ol>
	</li>
	<li>Project task instructions onto the participant&#8217;s screen: ask if he/she has any questions after reading them, and if he/she is ready to start the experiment.</li>
	<li>Ask the participant to solve 10 Newtonian physics problems.</li>
	<li>If needed, perform an impedance check during one of the 45 s breaks (not before problem 5).</li>
	<li>Make sure the participant takes the full 45&#160;s break before each problem (to determine the baseline).</li>
</ol>
4. End of Data Collection
<ol>
	<li>Stop data acquisition on all computers and remove the sensors from the participant.</li>
</ol>
5. After the Participant Has Left
<ol>
	<li>Clean the EEG cap with germicide and tidy up the equipment, according to the manufacturer&#8217;s recommendations. Save all the data files collected and create a backup on the FTP server.</li>
	<li>Fill in the participant spreadsheet: note any particular event or problem during data collection. Erase all cookies from the web browser.</li>
</ol>
6. Data Pre-processing and Export to the Integration Software
<ol start="1">
	<li>EEG
	<ol>
		<li>Import EEG data into EEG data analysis software:
		<ol>
			<li>Create three empty folders on the computer named &#8220;Raw data&#8221;, &#8220;History&#8221; and &#8220;Export&#8221; to paste the raw EEG data into the newly created Raw data file.</li>
			<li>In the EEG data analysis software, click &#8220;File/New Project &#8230;&#8221; and choose the raw data location by clicking Browse, then selecting the newly created raw data file. Choose the location of the &#8220;History&#8221; and &#8220;Export&#8221; folders in the same way.</li>
			<li>Click &#8220;OK&#8221;. (The window should contain all the participant&#8217;s EEG data).</li>
		</ol>
		</li>
		<li>Pre-process the brain signal:
		<ol>
			<li>Apply a filter and a notch (click &#8220;Transformations/IIR filters&#8230;&#8221;). In the window, enable the low cutoff at 1.5&#160;Hz with a slope of 12 dB and the high cutoff at 50&#160;Hz with a slope of 12&#160;dB. Also enable a notch at 60 Hz frequency.</li>
			<li>Because a DC amplifier is used, DC detrend the signal (Click &#8220;Transformations/DC Detrend&#8230;&#8221; and enable &#8220;based on time&#8221; at 100&#160;msec before marker and 100&#160;msec before DC connection).</li>
			<li>Perform a raw data inspection (Click &#8220;Transformation/Raw data inspection&#8230;&#8221; and select semi-automatic artifact removal). Select the following: maximal voltage 60&#160;&#181;V/ms; Max-min: 200 &#181;V in 200&#160;ms interval; amplitude: -400 to +400 &#181;V).</li>
			<li>Perform an automatic ICA with classic sphering for eye blink removal (myographic artifacts do not need to be removed because their range is outside of the frequencies of interest). (Click &#8220;Transformations/ICA&#8230;&#8221;. At the end of the ICA, process the inverse ICA.)</li>
			<li>Re-reference (&#8220;Transformations/Re-reference&#8230;&#8221;) the signal and select &#8220;common average&#8221;.</li>
			<li>Export (click &#8220;Export/Generic Data Export&#8230;&#8221;) the signal and markers in text format (Select the &#8220;.vhdr&#8221; box) for an eventual Matlab construction of the engagement index. Also select the &#8220;Write header file&#8221; and &#8220;Write marker file&#8221; boxes.</li>
		</ol>
		</li>
		<li>Import the signal in Matlab.
		<ol>
			<li>Start Matlab and type &#8220;eeglab&#8221; so the GUI of EEGLab appears and Import the data for one participant at a time. In the GUI, select item menu &#8220;File/Import Data/Using EEGLab functions and plugins/From Brain Vis Rec .vhdr file&#8221;.</li>
			<li>In the command window, paste a script16 that generates an engagement index. &#160; 
			NOTE: The cognitive engagement script is computed by the average of each Beta/(Alpha + Theta) ratio within a 20&#160;sec&#160;sliding window preceding time T. This procedure is repeated every second and a new sliding window is used to update the index.</li>
		</ol>
		</li>
		<li>In MS Excel, open the text file of the engagement index that is generated at the end of the script by Matlab and apply a z-score normalization on EEG data to allow intersubject comparison. (For each value, compute this formula in Excel: Z=(value &#8211; overall mean)/overall standard deviation.)</li>
		<li>Save the z-score engagement index signal in a CSV file in MS Excel. (Click File/Save as &#8230; and select CSV in the format type.)</li>
		<li>Repeat the procedure (from step 6.1.2.2.) for each participant.</li>
	</ol>
	</li>
	<li>Physiology:
	<ol>
		<li>Import EDA data in physiological data analysis software.</li>
		<li>Apply these parameters to pre-process the physiological signal:
		<ol>
			<li>Apply a logarithmic transformation to normalize the distribution of the conductance as per Venables and Christie&#8217;s23 method.</li>
			<li>Flat the signal on a 10&#160;sec sliding window24.</li>
		</ol>
		</li>
		<li>Within the physiological software, compute a z-score normalization on the EDA data to allow intersubject comparison. (Z=(value &#8211; overall mean)/overall standard deviation).
		<ol>
			<li>Highlight all the data with the cursor from the EDA channel.</li>
			<li>In the top menu, select the EDA channel, and select &#8220;mean&#8221; to obtain the mean value of the overall channel. Also select the EDA channel and &#8220;stddev&#8221; to obtain the standard deviation value of the overall channel.</li>
			<li>To compute the z-score equation, click &#8220;Transformation/Waveform Math &#8230;&#8221; and select the EDA channel in Source 1 . Select &#8220;&#8211;&#8221; (minus) in the mathematical operation window and select K in source 2. Select &#8220;New destination&#8221; in the destination menu and enter the mean value of the EDA channel (see step 6.2.3.2). Select &#8220;Transform entire wave&#8221;, click OK and click &#8220;Transformation /Waveform Math &#8230;&#8221;. Select the EDA-K channel in source 1, select &#8220;/&#8221; (divide) in the mathematical operation window, select K in source 2, select &#8220;New destination&#8221; in destination and enter the standard deviation value of the EDA channel (step 6.2.3.2). Select &#8220;Transform entire wave&#8221; and click OK.</li>
		</ol>
		</li>
		<li>Export the signal (arousal) in a CSV file. (Click File/Save as &#8230; and select CSV in the format type.)</li>
	</ol>
	</li>
	<li>Automatic facial emotion recognition:
	<ol>
		<li>Import video data from the media recorder into automatic facial emotion recognition software. (Click &#8220;File/New&#8230;/Participant&#8230;&#8221;. After selecting a new participant in the project menu by clicking on it, click &#8220;File/New/Analysis/Video&#8230;&#8221;. Click the magnifying glass next to Analysis 1 and choose the desired video file.
		<ol>
			<li>Select an offline analysis for &#8220;every third frame&#8221; and activate &#8220;continuous calibration&#8221;.</li>
			<li>Export valence data in a CSV file. (Click &#8220;Options/Settings/Logging&#8230;&#8221;, check the &#8220;Write valence value to the log file&#8221; box. Click &#8220;File/Export&#8230;&#8221;, choose the location where the log files will be exported, and check the &#8220;Save detailed log&#8221; box.)</li>
			<li>Open the CSV file in MS Excel. Copy the valence data column in a single column of SPSS software. Click &#8220;Analyse/Descriptives Statistics/Descriptives&#8221; and select the just-pasted variable name. Check the box &#8220;Save standardized values in variables&#8221;. A column with a z-score will appear. Copy-paste these z-scores over the old data in the Excel file.</li>
		</ol>
		</li>
		<li>Save the Excel file with the z-scores of the signal (valence) in CSV format.</li>
	</ol>
	</li>
</ol>
7. Data Integration and Synchronization
<ol start="1">
	<li>In behavioral analysis software:
	<ol>
		<li>Import eye-tracking videos (behavioral engagement). (Click &#8220;File/Import/Video in a New Observation&#8230;&#8221;. Name the new observation and choose the desired video file.)</li>
		<li>Code each video with pertinent behaviors and contextual events (time markers, right/wrong answers).</li>
		<li>Import all external data with the appropriate header: z-score of EEG signal (cognitive engagement), z-score of EDA signal (emotional engagement), z-score of valence data (emotional engagement). (Click &#8220;File/Import/External Data&#8230;&#8221;. Select the appropriate file type and select the correct CSV file.)</li>
	</ol>
	</li>
	<li>Synchronize time between computers according to these formulas:
	<ol>
		<li>Time in eye gaze from time in EEG = Time in eye gaze + second marker in EEG &#8211; first marker in eye gaze.</li>
		<li>Time in eye gaze from time in facial emotion recognition = Time in eye gaze + first marker in facial emotion &#8211; first marker in eye gaze.</li>
		<li>Time in eye gaze from time in electrodermal activity = Time in eye gaze + first marker in electrodermal activity &#8211; first marker in eye gaze.</li>
	</ol>
	</li>
	<li>Enter the offset data by pressing &#8220;Ctrl + Shift + =&#8221;, to open the Offset menu. Select &#8220;Numerical offset&#8221; to enter the time in seconds between each pair of data sources [OK?]), according to the calculations above.</li>
	<li>Generate a report according to the variables of interest in the study.
	<ol>
		<li>Select interesting variables that will be generated in the report (click &#8220;Analyze/Select Data/New Profile Data&#8230;&#8221;). From the left, slide the desired variables between the &#8220;Start&#8221; box and the &#8220;Results&#8221; box, on the right.</li>
		<li>Generate the report. (Click &#8220;Analyze/Numerical Analysis/New&#8230;&#8221;, click &#8220;Statistics&#8221; and check the mean box in the external data menu. Finish by clicking &#8220;Calculate&#8221;.)</li>
	</ol>
	</li>
	<li>Export the data into statistical analysis software and perform analysis according to the study objectives.</li>
</ol>

<ol>
	<li>Fredricks, J. A., Blumenfeld, P. C., Paris, A. H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=School+engagement:+Potential+of+the+concept,+state+of+the+evidence.">School engagement: Potential of the concept, state of the evidence.</a> Rev. Educ. Res. 74 (1), 59-109 (2004).</li><li>Clark, R. C., Mayer, R. E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> E-learning and the Science of Instruction. , (2011).</li><li>Zhang, D., Zhou, L., Briggs, R. O., Nunamaker, J. F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Instructional+video+in+e-learning:+Assessing+the+impact+of+interactive+video+on+learning+effectiveness.">Instructional video in e-learning: Assessing the impact of interactive video on learning effectiveness.</a> Inform. Manage. 43 (1), 15-27 (2006).</li><li>Freeman, F. G., Mikulka, P. J., Prinzel, L. J., Scerbo, M. W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Evaluation+of+an+adaptive+automation+system+using+three+EEG+indices+with+a+visual+tracking+task.">Evaluation of an adaptive automation system using three EEG indices with a visual tracking task.</a> Biol. Psychol. 50, 61-76 (1999).</li><li>Glimcher, P., Rustichini, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Neuroeconomics:+The+consilience+of+brain+and+decision.">Neuroeconomics: The consilience of brain and decision.</a> Science. 306 (5695), 447-452 (2004).</li><li>Lieberman, M. D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Social+Cognitive+neuroscience:+A+review+of+core+processes.">Social Cognitive neuroscience: A review of core processes.</a> Annu. Rev. Physiol. 58, 259-289 (2007).</li><li>Pope, A. T., Bogart, E. H., Bartolome, D. S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Biocybernetic+system+evaluates+indices+of+operator+engagement+in+automated+task.">Biocybernetic system evaluates indices of operator engagement in automated task.</a> Biol. Psychol. 40 (1-2), 187-195 (1995).</li><li>Mandryk, R., Inkpen, K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Physiological+indicators+for+the+evaluation+of+co-located+collaborative+play.">Physiological indicators for the evaluation of co-located collaborative play.</a> , (2004).</li><li>Allaire-Duquette, G., Charland, P., Riopel, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=At+the+very+root+of+the+development+of+interest:+Using+human+body+contexts+to+improve+women&#8217;s+emotional+engagement+in+introductory+physics.">At the very root of the development of interest: Using human body contexts to improve women&#8217;s emotional engagement in introductory physics.</a> Eur. J. Phy. Ed. 5 (2), 31-48 (2014).</li><li>Bradley, M. M., Lang, P. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Measuring+emotion:+The+self-assessment+manikin+and+the+semantic+differential.">Measuring emotion: The self-assessment manikin and the semantic differential.</a> J. Behav. Ther. Exp. Psy. 25 (1), 49-59 (1994).</li><li>Lang, P. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+emotion+probe:+Studies+of+motivation+and+attention.">The emotion probe: Studies of motivation and attention.</a> Am. Psychol. 50 (5), 372-385 (1995).</li><li>Ekman, P., Felt Friesen, W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=false,+and+miserable+smiles.">false, and miserable smiles.</a> J. Nonverbal Behav. 6 (4), 238-252 (1982).</li><li>Van Kuilenburg, H., Den Uyl, M. J., Isra&#235;l, M. L., Ivan, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Advances+in+face+and+gesture+analysis.">Advances in face and gesture analysis.</a> , 371-372 (2008).</li><li>Cacioppo, J., Tassinary, L. G., Berntson, G. G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Handbook of Psychophysiology. , (2007).</li><li>Lubar, J. F., Swartwood, M. O., Swartwood, J. N., O&#8217;Donnell, P. H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Evaluation+of+the+effectiveness+of+EEG+neurofeedback+training+for+ADHD+in+a+clinical+setting+as+measured+by+changes+in+T.O.V.A.+scores,+behavioral+ratings,+and+WISC+R+performance.">Evaluation of the effectiveness of EEG neurofeedback training for ADHD in a clinical setting as measured by changes in T.O.V.A. scores, behavioral ratings, and WISC R performance.</a> Biofeedback Self-reg. 20 (1), (1995).</li><li>Mikulka, P. J., Freeman, F. G., Scerbo, M. W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Effects+of+a+biocybernetic+system+on+the+vigilance+decrement.">Effects of a biocybernetic system on the vigilance decrement.</a> Hum. factors. 44 (4), 654-664 (2002).</li><li>Freeman, F. G., Mikulka, P. J., Scerbo, M. W., Scott, L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=An+evaluation+of+an+adaptive+automation+system+using+a+cognitive+vigilance+task.">An evaluation of an adaptive automation system using a cognitive vigilance task.</a> Biol. Psychol. 67 (3), 283-297 (2004).</li><li>Chaouachi, M., Chalfoun, P., Jraidi, I., Frasson, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Affect+and+mental+engagement:+Toward+adaptability+for+intelligent+systems.">Affect and mental engagement: Toward adaptability for intelligent systems.</a> Proceedings of the Twenty-Third International Florida Artificial Intelligence Research Society Conference (FLAIRS 2010), Association for the Advancement of Artificial Intelligence. , 355-360 (2010).</li><li>Courtemanche, F., Ortiz de Guinea, A., Titah, R., Fredette, M., Labont&#233;-LeMoyne, &. #. 2. 0. 1. ;. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Precision+is+in+the+eye+of+the+beholder:+Application+of+eye+fixation-related+potentials+to+information+systems+research.">Precision is in the eye of the beholder: Application of eye fixation-related potentials to information systems research.</a> J. Assoc. Inf. Syst. 15 (10), 651-678 (2014).</li><li>Courtemanche, F., Ortiz de Guinea, A., Titah, R., Fredette, M., Labont&#233;-Lemoyne, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Applying+eye+fixation-related+potentials+to+information+systems+research:+Demonstration+of+the+method+during+natural+IS+use+and+guidelines+for+research.">Applying eye fixation-related potentials to information systems research: Demonstration of the method during natural IS use and guidelines for research.</a> J. Assoc. Inf. Syst. 15 (10), (2014).</li><li>Courtemanche, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Un outil d&#8217;&#233;valuation neurocognitive des interactions humain-machine Doctoral thesis. , (2014).</li><li>Charland, P., L&#233;ger, P. M., Mercier, J., Skelling-Desmeules, Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Assessing+multiple+dimensions+of+learner+engagement+during+science+problem+solving+using+psychophysiological+and+behavioral+measures.">Assessing multiple dimensions of learner engagement during science problem solving using psychophysiological and behavioral measures.</a> , (2014).</li><li>Venables, P. H., Christie, M. J., Martin, I., Venables, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Electrodermal+activity.">Electrodermal activity.</a> Techniques in Psychophysiology. , 3-67 (1980).</li><li>Boucsein, W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Electrodermal Activity. , (2012).</li><li>Sarlo, M., Palomba, D., Buodo, G. M., Minghetti, R., Stegagno, L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Blood+pressure+changes+highlight+gender+differences+in+emotional+reactivity+to+arousing+pictures.">Blood pressure changes highlight gender differences in emotional reactivity to arousing pictures.</a> Biol. Psychol. 70 (3), 188-196 (2005).</li><li>Martens, R. L., Gulikers, J., Bastiaens, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+impact+of+intrinsic+motivation+on+e-learning+in+authentic+computer+tasks.">The impact of intrinsic motivation on e-learning in authentic computer tasks.</a> J. Comput. Assist. Lear. 20 (5), 368-376 (2004).</li></ol>

The authors have nothing to disclose.

In terms of critical steps within the protocol, it should first be pointed out that data quality is always the main focus for neurophysiological collection techniques. In this methodology, research assistants must pay special attention to instructing the subjects to minimize head movements that will interfere with valence monitoring (losing correct face angle for the camera) or generate myographic artifacts in the EEG. On the other hand, a balance must be maintained between the authenticity of real problem solving and interventions made for more ergonomic data collection. It is also important to note that EEG data collection is subject to electromagnetic fluctuations in the environment. Traditional EEG facilities try to isolate their apparatus from electromagnetic fluctuations with Faraday cages. However, because some of the equipment used in this methodology would generate electromagnetic fluctuations (mainly the eye-tracking device) inside the Faraday cage, this approach would be ineffective. We overcome the electromagnetic issues by paying particular attention to grounding and shielding all electrical devices.
As for modifications and troubleshooting with the technique, the initial synchronization strategy relied on the synchronization software&#8217;s capacity to precisely &#8220;start&#8221; data collection on multiple computers and programs together. Because critical and inconsistent delays between computers and programs were observed, post-collection resynchronization became necessary. Consequently, a syncbox device was added to the architecture. The syncbox sends a TTL marker to all the computers and programs that collect data. Synchronization becomes a matter of calculating the delay between the first syncbox markers.
One limitation of the technique that needs to be mentioned is the precision of signal analysis, which is limited by the cognitive engagement index. Because of the basic assumptions of the FFT, this index is generated on a 1&#160;sec epoch basis: the cognitive engagement script generates a value every second. In this paradigm, which focuses on authentic problem solving, this timeframe is acceptable, but more precise studies of engagement might encounter some limitations with this timeframe for analysis.
With respect to existing/alternative methods, it must be noted that emotional valence can also be derived with blood volume pressure18, 25 sensors. This technique could also be integrated into future research to evaluate its accuracy compared to the valence signal from facial emotion recognition software. We should also mention that the cognitive engagement index used in this study is a well-known one that has been used in previous published research. Some manufacturers of lightweight EEG devices claim to provide a similar measure, but it is difficult to assess the quality of the raw and processed data since their algorithms are unpublished.
Finally, this technique presents many possible applications in different fields. Of course, it will be of value in the field of education. Among other possibilities, this engagement assessment technique could be a powerful tool to inform course designers. For example, as Martens, Gulikers and Bastiaens26 observed, &#8220;quite often, developers tend to add multimedia add-ons, simulations, and so on, mainly because technology makes it possible, even though they are not based on careful educational analysis and design.&#8221; Thus, neurophysiological data could inform designers if a specific add-on is valuable, if the content is too complex, if the proposed learning strategies are efficient, etc. In addition, real-time assessment of learner engagement opens up possibilities for adaptive e-learning or e-assessment environments. We can foresee a learner, wearing a lightweight EEG helmet, being warned by the system when his/her engagement level is declining and, for example, prompted to pause or react accordingly. It would also be possible to develop adaptive assessment tasks, based on engagement indexes. A fair amount of research and development are currently being conducted in the innovative field of brain-computer interfaces (BCI).

Engagement plays a crucial role in learning. For Clark and Mayer2, &#8220;all learning requires engagement,&#8221; regardless of delivery media. Zhang et al.3 also suggested that increased student engagement can improve learning outcomes, such as problem solving and critical thinking skills. Defining engagement remains a challenge. In their literature review, Fredricks, Blumenfeld and Paris1 defined engagement by its multifaceted nature: &#8220;Behavioural engagement draws on the idea of participation; it includes involvement in academic and social or extracurricular activities. (&#8230;) Emotional engagement encompasses positive and negative reactions to teachers, classmates, academics, and school and is presumed to create ties to an object and influence willingness to do the work. Finally, cognitive engagement draws on the idea of mental investment; it incorporates thoughtfulness and willingness to exert the effort necessary to comprehend complex ideas and master difficult skills.&#8221;
Fredricks, Blumenfeld and Paris1 also claimed that a focus on behavior, emotion, and cognition, within the concept of engagement, may provide a richer characterization of learning. These authors pointed out that a robust body of research addresses each component of engagement separately, but these components had not been studied in conjunction. They also observed that little information is available about interactions between the dimensions and that more studies could contribute to planning finely tuned teaching interventions. As a step in that direction, this paper describes a research methodology that was developed to gather and analyze quantitative and qualitative data, synchronously, on behavioral, emotional and cognitive engagement during learning tasks.
Bringing the Neurosciences into Education
Behavior, and consequently behavioral engagement, has long been the central focus of studies in education: research designs focused mainly on changes in knowledge and behavior occurring over long periods of time, between pre- and post-tests, and over intervals of hours, weeks, months or years. Discriminating between behavioral, emotional, and cognitive engagement remains a challenge because the last two dimensions are not systematically observable externally. Cognition and emotions must either be inferred from observations or assessed with self-report measures. From an external point of view, it remains difficult to determine whether students are trying to get their work done as quickly as possible or using deep-level learning strategies to master a specific content. In point of fact, Fredricks, Blumenfeld and Paris1 were unable to find any published studies using direct, objective measures of cognitive engagement.
Recent technological developments in the field of neurosciences have created new possibilities for research in education. New data collection methods and analysis algorithms developed in the field of neuroergonomics seem very promising for qualitative and quantitative studies during learning tasks. Other disciplines, such as economics, psychology, marketing, and ergonomics, have been using neurophysiological measurements to assess cognitive engagement for some time4-8. Neurophysiological measures, coupled with efficient analysis algorithms, allow one to study a phenomenon without disturbing it. By their nature, self-report questionnaires disengage students from learning. Neurophysiological measures allow research designs to be carried out in more authentic learning environments. These tools include equipment to monitor heart rate, breathing rate, blood pressure, body temperature, pupil diameter, electrodermal activity, electroencephalography (EEG), etc.
Gathering Synchronized Data on Behavioral, Emotional, and Cognitive Engagement
As representative outcomes following the use of this protocol, this paper will present partial results of a study in which learners had to solve, on a computer screen, ten problems in mechanical physics. These problems were developed in previous work9. Neurophysiological data were collected while the learners were solving the problems and relaxing during a 45&#160;s break, with their eyes closed, after each problem.
As mentioned above, behavioral engagement data consist of software interactions (mouse movements and clicks), eye gaze, performance and answers to questions produced by a learner interacting with the system while accomplishing the task1. An eye-tracking system was used to collect software interactions and eye gaze data. Performance data (time to solve a problem, correctness of answers) were collected on a survey website that was used to present the task. This website was also used to gather self-report data collected with a questionnaire adapted from Bradley and Lang10. Emotional engagement involves characterization of emotions. According to Lang11, emotions are characterized in terms of valence (pleasant/unpleasant) and arousal (calm/aroused). Emotional engagement data were accordingly collected, using automatic facial emotion recognition software that quantifies emotional valence and an electrodermal activity encoder/sensor for arousal12,13. Electrodermal activity (EDA) refers to the recorded electrical resistance between two electrodes when a very weak electrical current is steadily passed between them. Cacioppo, Tassinary and Berntson14 showed that the resistance recorded varies according to the subject&#8217;s arousal. Thus, psychophysiological data, such as valence or arousal, are considered as correlates of emotional engagement.
Finally, cognitive engagement data are collected through electroencephalography (EEG). EEG measures, on the scalp, the synchronized electrical activity of groups of neurons in the brain. Electrical signals recorded from the scalp are often oscillatory and composed of frequency components. By convention, these frequencies are grouped in sequences, known as bands. For example, alpha, beta and theta bands are the focus of this study. According to neuroscientific studies14, these bands reflect different cognitive processing abilities in specific areas of the brain. Thus, the analysis of the power spectral density (PSD) of specific frequencies, combined with numerous studies7,15 on alertness and attention, allows researchers to quantify cognitive engagement during a task. As Mikulka et al.16 noted, research has shown a direct relationship between beta activity and cognitive alertness and an indirect relationship between alpha and theta activity and alertness. Thus, Pope, Bogart and Bartoleme7 developed an engagement index that computes the PSD of three bands: beta / (alpha + theta). This ratio was validated in other studies on engagement16,17,18. To characterize cognitive engagement over time, a fast Fourier transform (FFT) converts the EEG signal from each active site (F3, F4, O1, O2) into a power spectrum. The EEG engagement index at time T is computed by the average of each engagement ratio within a 20&#160;sec&#160;sliding window preceding time T. This procedure is repeated every second and a new sliding window is used to update the index.
Since the aim of this methodology is to provide a rich analysis of the multiple dimensions of engagement, data synchronization is crucial. As Leger et al.19 remind readers, equipment manufacturers strongly recommend using only one computer per measurement tool to guarantee their specified precision level. Thus, when multiple computers are employed, synchronization between recording computers becomes a critical step. The recordings cannot all be started at the exact same time, and each data stream has its specific time frame (e.g., sec&#160;0 of eye tracking &#8800; sec&#160;0 of EEG or physiological data). This is extremely important: desynchronization between data streams means errors in the quantification of each dimension of engagement. There are different ways of synchronizing concurrent physiological and behavioral recordings. These methods may be divided into two main approaches; direct and indirect20. The protocol presented in the next section is based on an indirect approach where an external device, a syncbox, is used to send transistor-transistor logic (TTL) signals to all the recording equipment (as shown in Figure 1). As each piece of equipment has a different start time, the TTL markers are recorded in the log files with a relative delay. Markers are then used to realign the signals and thus ensure proper synchronization after each recording. A behavioral analysis software program that allows external file integration is used to re-synchronize the timeline of each data stream and to perform quantitative and qualitative analysis of each dimension of engagement.
<img alt="Figure 1" src="/files/ftp_upload/52627/52627fig1.jpg" /> 
Figure 1. Architecture of the Data Collection System. The lab environment in which behavioral (eye-tracking), emotional (EDA and facial emotion) and cognitive (EEG) engagement data are collected contains many computers. This raises a synchronization challenge for data that are referenced on their respective computer clocks. To be able to analyze all data in the same reference time, the lab setup involves a syncbox that sends TTL signals to all data streams. <a href="https://www.jove.com/files/ftp_upload/52627/52627fig1large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
To evaluate the precision of the methodology in terms of synchronization, 45&#160;sec&#160;pauses were introduced before each of the mechanical physics problems. During these pauses, subjects had to relax and to close their eyes. As seen in other studies4,9,16,17,18, these pauses should induce significant variations in the collected signal: the two eye pupil dots in eye-tracking immediately disappear (behavioral engagement) and an immediate drop in cognitive engagement (EEG signal) is observed. These specific components of the signal are used to evaluate the general validity of the synchronization. The recent publication of papers that fully or partially rely on this synchronization procedure, in the fields of information systems19, human-machine interactions21 and education9, 22, provides evidence of its effectiveness.

<table><tbody><tr><td>EGI GSN-32&amp;nbsp;</td><td>EGI</td><td></td><td>Dense array EEG</td></tr><tr><td>Netstation v.5.0</td><td>EGI</td><td></td><td>EEG data collection software: EEG is collected with 32-electrode dense array electroencephalography (dEEG) geodesic sensor net using Netstation acquisition software and EGI amplifiers (Electrical Geodesics, Inc). The vertex (recording site Cz) is the reference electrode for recording. Impedance is kept below 50 k&amp;Omega; with a sampling rate of 250 Hz.&amp;nbsp;</td></tr><tr><td>Facereader v.4</td><td>Noldus</td><td></td><td>Facial emotion recognition software</td></tr><tr><td>Syncbox</td><td>Noldus</td><td></td><td>Syncbox start the co-registration of EEG and gaze data by sending a Transistor-Transistor Logic (TTL) signal to the EGI amplifier and a keystroke signal to the Tobii Studio v 3.2.</td></tr><tr><td>Logitech C600&amp;nbsp; Webcam</td><td></td><td>960-000396</td><td>Webcam used to gather video data sent to mediarecorder and that will be analyzed in Facereader</td></tr><tr><td>The Observer XT</td><td>Noldus</td><td></td><td>Integration and synchronization software: The Noldus Observer XT (Noldus Information Technology) is used to synchronize all behavioral, emotional and cognitive engagement data.&amp;nbsp;</td></tr><tr><td>On-Screen LED illumination</td><td>Noldus</td><td></td><td>Neon positioned on computer screen in order to correctly light the face of subjects</td></tr><tr><td>MediaRecorder</td><td>Noldus</td><td></td><td>Video data collection software</td></tr><tr><td>Tobii 60X</td><td>Tobii</td><td></td><td>Collect eye-movement patterns :&amp;nbsp; used to record subjects&amp;rsquo; eye movement patterns at 60Hz during the experiment.&amp;nbsp;</td></tr><tr><td>Tobii Studio v.3.2</td><td>Tobii</td><td></td><td>Eye-tracking data collection and analysis software</td></tr><tr><td>Analyzer 2</td><td>Brainvision</td><td></td><td>EEG signal processing software</td></tr><tr><td>Acqknowledge v.4.0</td><td>Biopac</td><td>ACK100M</td><td>Physiological signal acquisition and processing software</td></tr><tr><td>Control III germicide solution</td><td>Maril Products.</td><td>10002REVA-20002-1</td><td>Disinfectant solution used with EEG helmets : recommended by EGI</td></tr><tr><td>Unipark</td><td>QuestBack AG</td><td></td><td>Online survey environment</td></tr></tbody></table>

assessing the multiple dimensions of engagement to characterize learning: a neurophysiological perspective

In a recent theoretical synthesis on the concept of engagement, Fredricks, Blumenfeld and Paris1 defined engagement by its multiple dimensions: behavioral, emotional and cognitive. They observed that individual types of engagement had not been studied in conjunction, and little information was available about interactions or synergy between the dimensions; consequently, more studies would contribute to creating finely tuned teaching interventions. Benefiting from the recent technological advances in neurosciences, this paper presents a recently developed methodology to gather and synchronize data on multidimensional engagement during learning tasks. The technique involves the collection of (a) electroencephalography, (b) electrodermal, (c) eye-tracking, and (d) facial emotion recognition data on four different computers. This led to synchronization issues for data collected from multiple sources. Post synchronization in specialized integration software gives researchers a better understanding of the dynamics between the multiple dimensions of engagement. For curriculum developers, these data could provide informed guidelines for achieving better instruction/learning efficiency. This technique also opens up possibilities in the field of brain-computer interactions, where adaptive learning or assessment environments could be developed.

Figures 2 and 3 show screenshots of the results of the integration and synchronization of behavioral, emotional and cognitive engagement data in a behavioral analysis software application. In both figures, the left-hand section organizes the research subjects and the coding scheme. In the middle section, a video (with red dots) shows the subject&#8217;s eye gaze during the task. The subject&#8217;s behavioral engagement can be inferred based on what he/she is looking at during the task and what actions are taken. In the lower section, a time marker is synchronously scrolling in three tracks of data: the EDA (arousal) and facial emotion valence for emotional engagement and the EEG engagement index for cognitive engagement. When data are collected from all the subjects, the software also provides basic descriptive statistics that can eventually be used to perform intersubject analysis in other statistical analysis software.
<img alt="Figure 2" src="/files/ftp_upload/52627/52627fig2.jpg" /> 
Figure 2. Multidimensional Engagement Data at the Beginning of a Problem-solving Task. A screenshot of a subject at the beginning of a problem-solving phase. The learner is reading the introduction to the problem: the eye gaze is on the third line. At this time (the red line represents a time cursor), the subject&#8217;s arousal has just passed a peak of anticipation of the problem to be solved but is still high compared to baseline, emotional valence seems neutral, and EEG cognitive engagement seems at its maximum. <a href="https://www.jove.com/files/ftp_upload/52627/52627fig2large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 3" src="/files/ftp_upload/52627/52627fig3.jpg" /> 
Figure 3. Multidimensional Engagement Data During a Pause in the Task. Data from a pause before a problem-solving task. This pause is useful to establish the subject&#8217;s baseline just before the task. Here, because the subject&#8217;s eyes are closed, the valence data are not available. Cognitive engagement (EEG signal) is rising slightly from its minimum. The subject is slowly re-engaging with the task, anticipating the end of the pause. Arousal (EDA signal) is constantly declining. <a href="https://www.jove.com/files/ftp_upload/52627/52627fig3large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>

Watch this Scientific Journal Video about Assessing the Multiple Dimensions of Engagement to Characterize Learning: A Neurophysiological Perspective at JoVE.com

Assessing the Multiple Dimensions of Engagement to Characterize Learning: A Neurophysiological Perspective

This paper aims to describe the techniques involved in the collection and synchronization of the multiple dimensions (behavioral, affective and cognitive) of learners&#8217; engagement during a task.

In a recent theoretical synthesis on the concept of engagement, Fredricks, Blumenfeld and Paris1 defined engagement by its multiple dimensions: ...

assessing-multiple-dimensions-engagement-to-characterize-learning

Nanyang Technological University

Research

JoVE Journal

Behavior

12.4K Views.  Université du Québec à Montréal. This paper aims to describe the techniques involved in the collection and synchronization of the multiple dimensions (behavioral, affective and cognitive) of learners’ engagement during a task. 

Video: Assessing the Multiple Dimensions of Engagement to Characterize Learning: A Neurophysiological Perspective

Combining Computer Game-Based Behavioural Experiments With High-Density EEG and Infrared Gaze Tracking

Experimental paradigms are valuable insofar as the timing and other parameters of their stimuli are well specified and controlled, and insofar as they yield data relevant to the cognitive processing that occurs under ecologically valid conditions. These two goals often are at odds, since well controlled stimuli often are too repetitive to sustain subjects' motivation. Studies employing electroencephalography (EEG) are often especially sensitive to this dilemma between ecological validity and experimental control: attaining sufficient signal-to-noise in physiological averages demands large numbers of repeated trials within lengthy recording sessions, limiting the subject pool to individuals with the ability and patience to perform a set task over and over again. This constraint severely limits researchers' ability to investigate younger populations as well as clinical populations associated with heightened anxiety or attentional abnormalities. Even adult, non-clinical subjects may not be able to achieve their typical levels of performance or cognitive engagement: an unmotivated subject for whom an experimental task is little more than a chore is not the same, behaviourally, cognitively, or neurally, as a subject who is intrinsically motivated and engaged with the task. A growing body of literature demonstrates that embedding experiments within video games may provide a way between the horns of this dilemma between experimental control and ecological validity. The narrative of a game provides a more realistic context in which tasks occur, enhancing their ecological validity (Chaytor &amp; Schmitter-Edgecombe, 2003). Moreover, this context provides motivation to complete tasks. In our game, subjects perform various missions to collect resources, fend off pirates, intercept communications or facilitate diplomatic relations. In so doing, they also perform an array of cognitive tasks, including a Posner attention-shifting paradigm (Posner, 1980), a go/no-go test of motor inhibition, a psychophysical motion coherence threshold task, the Embedded Figures Test (Witkin, 1950, 1954) and a theory-of-mind (Wimmer &amp; Perner, 1983) task. The game software automatically registers game stimuli and subjects' actions and responses in a log file, and sends event codes to synchronise with physiological data recorders. Thus the game can be combined with physiological measures such as EEG or fMRI, and with moment-to-moment tracking of gaze. Gaze tracking can verify subjects' compliance with behavioural tasks (e.g. fixation) and overt attention to experimental stimuli, and also physiological arousal as reflected in pupil dilation (Bradley et al., 2008). At great enough sampling frequencies, gaze tracking may also help assess covert attention as reflected in microsaccades - eye movements that are too small to foveate a new object, but are as rapid in onset and have the same relationship between angular distance and peak velocity as do saccades that traverse greater distances. The distribution of directions of microsaccades correlates with the (otherwise) covert direction of attention (Hafed &amp; Clark, 2002).

Procedures for recording high-density EEG and gaze data during computer game-based cognitive tasks are described. Using a video game to present cognitive tasks enhances ecological validity without sacrificing experimental control.

Experimental paradigms are valuable insofar as the timing and other parameters of their stimuli are well specified and controlled, and insofar as they ...

Neuroscience

A Cross-Disciplinary and Multi-Modal Experimental Design for Studying Near-Real-Time Authentic Examination Experiences

Over the past ten years, research into students&#39; emotions in educational environments has increased. Although researchers have called for more studies that rely on objective measures of emotional experience, limitations on utilizing multi-modal data sources exist. Studies of emotion and emotional regulation in classrooms traditionally rely on survey instruments, experience-sampling, artifacts, interviews, or observational procedures. These methods, while valuable, are mainly&#160;dependent on participant or observer subjectivity and is limited in its authentic measurement of students&#39; real-time performance to a classroom activity or task. The latter, in particular, poses a stumbling block to many scholars seeking to objectively measure emotions and other related measures in the classroom, in real-time.
The purpose of this work is to present a protocol to experimentally study students&#39; real-time responses to exam experiences during an authentic assessment situation. For this, a team of educational psychologists, engineers, and engineering education researchers designed an experimental protocol that retained the limits required for accurate physiological sensor measurement, best-practices of salivary collection, and an authentic testing environment. In particular, existing studies that rely on physiological sensors are conducted in experimental environments that are disconnected from educational settings (e.g., Trier Stress Test), detached in time (e.g., before or after a task), or introduce analysis error (e.g., use of sensors in environments where students are likely to move). This limits our understanding of students&#39; real-time responses to classroom activities and tasks. Furthermore, recent research has called for more considerations to be covered around issues of recruitment, replicability, validity, setups, data cleaning, preliminary analysis, and particular&#160;circumstances (e.g., adding a variable in the experimental design) in academic emotions research that relies on multi-modal approaches.

An experimental design was developed to investigate the real-time influences of an examination experience to assess the emotional realities students experience in higher education settings and tasks. This design is the result of a cross-disciplinary (e.g., educational psychology, biology, physiology, engineering) and multi-modal (e.g., salivary markers, surveys, electrodermal sensor) approach.

Over the past ten years, research into students&#39; emotions in educational environments has increased. Although researchers have called for more studies ...

Multimodal Protocol for Assessing Metacognition and Self-Regulation in Adults with Learning Difficulties

Learning disabilities (LDs) encompass disorders of those who have difficulty learning and using academic skills, exhibiting performance below expectations for their chronological age in the areas of reading, writing, and/or mathematics. Each of the disorders making up the LDs involve different deficits; however, some commonalities can be found within that heterogeneity, such in terms of learning self-regulation and metacognition. Unlike in early ages and later educational levels, there are hardly any evidence-based evaluation protocols for adults with LDs. LDs influence academic performance but also have serious consequences in professional, social, and family contexts. In response to this, the current work proposes a multimodal evaluation protocol focused on metacognitive, self-regulation of learning, and emotional processes, which make up the basis of the difficulties in adults with LDs. The assessment is carried out through analysis of the on-line learning process using a variety methods, techniques, and sensors (e.g., eye tracking, facial expressions of emotion, physiological responses, concurrent verbalizations, log files, screen recordings of human-machine interactions) and off-line methods (e.g., questionnaires, interviews, and self-report measures). This theoretically-driven and empirically-based guideline aims to provide an accurate assessment of LDs in adulthood in order to design effective prevention and intervention proposals.

The current work proposes a multimodal evaluation protocol focused on metacognitive, self-regulation of learning, and emotional processes, which make up the basis of the difficulties in adults with LDs.

Learning disabilities (LDs) encompass disorders of those who have difficulty learning and using academic skills, exhibiting performance below expectations ...

Measuring Engagement of Spectators of Social Digital Games

The goal of this methodology is to assess explicit and implicit measures of engagement of spectators during social digital games in a group of participants with motion tracking systems. In the context of games that are not confined within a screen, measuring the different dimensions of engagement such as physiological arousal can be challenging. The focus of the study is made on the spectators of the game and the differences in their engagement according to interactivity. Engagement is measured with physiological and self-reported arousal, as well as an engagement questionnaire at the end of the experiment. Physiological arousal is measured with electrodermal activity (EDA) sensors that record the data on a portable device (EDA box). Portability was essential because of the nature of the game, which is akin to a life-size pong and includes many participants that move. To have an overview of the events of the game, three cameras are used to film three angles of the playing field. To synchronize the EDA data with events happening in the game, boxes with digital numbers are used and put in the frames of cameras. Signals are sent from a sync box simultaneously to the EDA boxes and to light boxes. The light boxes show the synchronization numbers to the cameras, and the same numbers are also logged on the EDA data file. That way, it is possible to record EDA of many people that move freely in a large space and synchronize this data with events in the game. In our particular study, we were able to assess the differences in arousal for the different conditions of interactivity. One of the limitations of this method is that the signals cannot be sent farther than 20 meters away. This method is, therefore, appropriate for recording physiological data in games with an unlimited number of players but is restricted to a limited space.

We propose a methodology that enables measuring engagement of spectators in a social digital game combining physiological and self-reported data. As this digital game involves a group of freely moving people, the experience is filmed using a synchronizing technique that links physiological data with events in the game.

The goal of this methodology is to assess explicit and implicit measures of engagement of spectators during social digital games in a group of ...

Eye-tracking Technology and Data-mining Techniques used for a Behavioral Analysis of Adults engaged in Learning Processes

Behavioral analysis of adults engaged in learning tasks is a major challenge in the field of adult education. Nowadays, in a world of continuous technological changes and scientific advances, there is a need for life-long learning and education within both formal and non-formal educational environments. In response to this challenge, the use of eye-tracking technology and data-mining techniques, respectively, for supervised (mainly prediction) and unsupervised (specifically cluster analysis) learning, provide methods for the detection of forms of learning among users and/or the classification of their learning styles. In this study, a protocol is proposed for the study of learning styles among adults with and without previous knowledge at different ages (18 to 69-year-old) and at different points throughout the learning process (start and end). Statistical analysis-of-variance techniques mean that differences may be detected between the participants by type of learner and previous knowledge of the task. Likewise, the use of&#160;unsupervised learning clustering techniques throws light on similar forms of learning among the participants across different groups. All these data will facilitate personalized proposals from the teacher for the presentation of each task at different points in the chain of information processing. It will likewise be easier for the teacher to adapt teaching materials to the learning needs of each student or group of students with similar characteristics.

We present a protocol for a behavioral analysis of adults (ages 18 to 70-year-old) engaged in learning processes, undertaking tasks designed for Self-Regulated Learning (SRL). The participants, university teachers and students, and adults from the University of Experience, were monitored with eye-tracking devices and the data were analyzed with data-mining techniques.

Behavioral analysis of adults engaged in learning tasks is a major challenge in the field of adult education. Nowadays, in a world of continuous ...

Enhancing Classroom Engagement: AI-Based Analysis of Student Attention Levels

Artificial Intelligence-Based System for Detecting Attention Levels in Students

The attention level of students in a classroom can be improved through the use of Artificial Intelligence (AI) techniques. By automatically identifying the attention level, teachers can employ strategies to regain students' focus. This can be achieved through various sources of information.
One source is to analyze the emotions reflected on students' faces. AI can detect emotions, such as neutral, disgust, surprise, sadness, fear, happiness, and anger. Additionally, the direction of the students' gaze can also potentially indicate their level of attention. Another source is to observe the students' body posture. By using cameras and deep learning techniques, posture can be analyzed to determine the level of attention. For example, students who are slouching or resting their heads on their desks may have a lower level of attention. Smartwatches distributed to the students can provide biometric and other data, including heart rate and inertial measurements, which can also be used as indicators of attention. By combining these sources of information, an AI system can be trained to identify the level of attention in the classroom. However, integrating the different types of data poses a challenge that requires creating a labeled dataset. Expert input and existing studies are consulted for accurate labeling. In this paper, we propose the integration of such measurements and the creation of a dataset and a potential attention classifier. To provide feedback to the teacher, we explore various methods, such as smartwatches or direct computers. Once the teacher becomes aware of attention issues, they can adjust their teaching approach to re-engage and motivate the students. In summary, AI techniques can automatically identify the students' attention level by analyzing their emotions, gaze direction, body posture, and biometric data. This information can assist teachers in optimizing the teaching-learning process.

Author Spotlight: Addressing Technical and Subjective Challenges in Measuring Classroom Attention

This paper proposes an artificial intelligence-based system to automatically detect whether students are paying attention to the class or are distracted. This system is designed to help teachers maintain students' attention, optimize their lessons, and dynamically introduce modifications in order for them to be more engaging.

The attention level of students in a classroom can be improved through the use of Artificial Intelligence (AI) techniques. By automatically identifying ...

Engineering

Measuring Statistical Learning Across Modalities and Domains in School-Aged Children Via an Online Platform and Neuroimaging Techniques

Statistical learning, a fundamental skill to extract regularities in the environment, is often considered a core supporting mechanism of the first language development. While many studies of statistical learning are conducted within a single domain or modality, recent evidence suggests that this skill may differ based on the context in which the stimuli are presented. In addition, few studies investigate learning as it unfolds in real-time, rather focusing on the outcome of learning. In this protocol, we describe an approach for identifying the cognitive and neural basis of statistical learning, within an individual, across domains (linguistic vs. non-linguistic) and sensory modalities (visual and auditory). The tasks are designed to cast as little cognitive demand as possible on participants, making it ideal for young school-aged children and special populations. The web-based nature of the behavioral tasks offers a unique opportunity for us to reach more representative populations nationwide, to estimate effect sizes with greater precision, and to contribute to open and reproducible research. The neural measures provided by the functional magnetic resonance imaging (fMRI) task can inform researchers about the neural mechanisms engaged during statistical learning, and how these may differ across individuals on the basis of domain or modality. Finally, both tasks allow for the measurement of real-time learning, as changes in reaction time to a target stimulus is tracked across the exposure period. The main limitation of using this protocol relates to the hour-long duration of the experiment. Children might need to complete all four statistical learning tasks in multiple sittings. Therefore, the web-based platform is designed with this limitation in mind so that tasks may be disseminated individually. This methodology will allow users to investigate how the process of statistical learning unfolds across and within domains and modalities in children from different developmental backgrounds.

Presented here is a protocol introducing a set of child-friendly statistical learning tasks geared towards examining children&#8217;s learning of temporal statistical patterns across domains and sensory modalities. The developed tasks collect behavioral data using the web-based platform and task-based functional magnetic resonance imaging (fMRI) data for examining neural engagement during statistical learning.

Statistical learning, a fundamental skill to extract regularities in the environment, is often considered a core supporting mechanism of the first ...

Inter-Brain Synchrony in Open-Ended Collaborative Learning: An fNIRS-Hyperscanning Study

fNIRS hyperscanning is widely used to detect the neurobiological underpinnings of social interaction. With this technique, researchers qualify the concurrent brain activity of two or more interactive individuals with a novel index called inter-brain synchrony (IBS) (i.e., phase and/or amplitude alignment of the neuronal or hemodynamic signals across time). A protocol for conducting fNIRS hyperscanning experiments on collaborative learning dyads in a naturalistic learning environment is presented here. Further, a pipeline of analyzing IBS of oxygenated hemoglobin (Oxy-Hb) signal is explained. Specifically, the experimental design, the process of NIRS data recording, data analysis methods, and future directions are all discussed. Overall, implementing a standardized fNIRS hyperscanning pipeline is a fundamental part of second-person neuroscience. Also, this is in line with the call for open-science to aid the reproducibility of research.

The protocol for conducting fNIRS hyperscanning experiments on collaborative learning dyads in a naturalistic learning environment is outlined. Further, a pipeline to analyze the Inter-Brain Synchrony (IBS) of oxygenated hemoglobin (Oxy-Hb) signals is presented.

fNIRS hyperscanning is widely used to detect the neurobiological underpinnings of social interaction. With this technique, researchers qualify the ...

Multimodal Large-Scale Neural Recordings and Analysis on HD-MEA 

Recording and Analyzing Multimodal Large-Scale Neuronal Ensemble Dynamics on CMOS-Integrated High-Density Microelectrode Array

Large-scale neuronal networks and their complex distributed microcircuits are essential to generate perception, cognition, and behavior that emerge from patterns of spatiotemporal neuronal activity. These dynamic patterns emerging from functional groups of interconnected neuronal ensembles facilitate precise computations for processing and coding multiscale neural information, thereby driving higher brain functions. To probe the computational principles of neural dynamics underlying this complexity and investigate the multiscale impact of biological processes in health and disease, large-scale simultaneous recordings have become instrumental. Here, a high-density microelectrode array (HD-MEA) is employed to study two modalities of neural dynamics - hippocampal and olfactory bulb circuits from ex-vivo mouse brain slices and neuronal networks from in-vitro cell cultures of human induced pluripotent stem cells (iPSCs). The HD-MEA platform, with 4096 microelectrodes, enables non-invasive, multi-site, label-free recordings of extracellular firing patterns from thousands of neuronal ensembles simultaneously at high spatiotemporal resolution. This approach allows the characterization of several electrophysiological network-wide features, including single/-multi-unit spiking activity patterns and local field potential oscillations. To scrutinize these multidimensional neural data, we have developed several computational tools incorporating machine learning algorithms, automatic event detection and classification, graph theory, and other advanced analyses. By supplementing these computational pipelines with this platform, we provide a methodology for studying the large, multiscale, and multimodal dynamics from cell assemblies to networks. This can potentially advance our understanding of complex brain functions and cognitive processes in health and disease. Commitment to open science and insights into large-scale computational neural dynamics could enhance brain-inspired modeling, neuromorphic computing, and neural learning algorithms. Furthermore, understanding the underlying mechanisms of impaired large-scale neural computations and their interconnected microcircuit dynamics could lead to the identification of specific biomarkers, paving the way for more accurate diagnostic tools and targeted therapies for neurological disorders.

Author Spotlight: Advancing Large-Scale Neural Dynamics Through HD-MEA Technology

Here, we employ HD-MEA to delve into computational dynamics of large-scale neuronal ensembles, particularly in hippocampal, olfactory bulb circuits, and human neuronal networks. Capturing spatiotemporal activity, combined with computational tools, provides insights into neuronal ensemble complexity. The method enhances understanding of brain functions, potentially identifying biomarkers and treatments for neurological disorders.

Large-scale neuronal networks and their complex distributed microcircuits are essential to generate perception, cognition, and behavior that emerge from ...

Perspectives on Neuropsychology

Source: Laboratories of Jonas T. Kaplan and Sarah I. Gimbel&#8212; University of Southern California
Neuropsychology is a complex field, as it investigates how mental processes are executed in the brain&#8212;events that integrate concepts from biochemistry, biology, psychology, and neuroscience. Although the multidisciplinary nature of neuropsychology prepares young learners for a variety of careers, it also poses a challenge in that it forces students to study concepts outside of their comfort zone. For example, a psychology major may have difficulty grasping neuroanatomy&#8212;a challenging topic in itself&#8212;given that the brain is a complicated, three-dimensional organ that is typically represented two-dimensionally in textbooks.
This JoVE collection in Neuropsychology introduces major concepts and methods in the field, and showcases how advances in imaging technology have allowed us to look inside the brain and visualize its structure and function. Importantly, these videos are also meant to reassure students that you don&#8217;t need to be an expert in all aspects of neuropsychology to understand how the brain shapes our experiences, behaviors, and social interactions in everyday life. For instance, the video "Decision-making and the Iowa Gambling Task" explores how damage to a specific region of the brain can affect an individual&#8217;s proclivity for risky decisions, like stealing.
These JoVE videos in Neuropsychology provide the perfect introduction to this complex topic, and also explore some of the potential, exciting future directions in this field. With further advances in technology, one day researchers might even be able to visualize dynamic changes in a person&#8217;s brain outside of a laboratory setting&#8212;for example, when they are sitting in a classroom listening to a lecture on neuropsychology.