Name: multi-modal signals for analyzing pain responses to thermal and electrical stimuli
Uploaded: 2019-04-05T13:00:00
Duration: 9 min 16 s
Description: Watch this Scientific Journal Video about Multi-Modal Signals for Analyzing Pain Responses to Thermal and Electrical Stimuli Video at JoVE.com

The authors would like to thank Verena Friedrich, Maria Velana, Sandra Gebhardt, Romy B&#228;rwaldt, and Tina Daucher for their precious help in conducting the study. In addition, a special thank you goes out to Dr. Stefanie Rukavina for her scientific support. This research was part of the DFG/TR233/12 (http://www.dfg.de/) &#8220;Advancement and Systematic Validation of an Automated Pain Recognition System on the Basis of Facial Expression and Psychobiological Parameters&#8221; project, funded by the German Research Foundation.

<ol>
	<li>Hawker, G. A., Mian, S., Kendzerska, T., French, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Measures+of+adult+pain:+Visual+analog+scale+for+pain+(vas+pain),+numeric+rating+scale+for+pain+(nrs+pain),+mcgill+pain+questionnaire+(mpq),+short&#8208;form+mcgill+pain+questionnaire+(sf&#8208;mpq),+chronic+pain+grade+scale+(cpgs),+short+form&#8208;36+bodily+pain+scale+(sf&#8208;36+bps),+and+measure+of+intermittent+and+constant+osteoarthritis+pain+(icoap).">Measures of adult pain: Visual analog scale for pain (vas pain), numeric rating scale for pain (nrs pain), mcgill pain questionnaire (mpq), short&#8208;form mcgill pain questionnaire (sf&#8208;mpq), chronic pain grade scale (cpgs), short form&#8208;36 bodily pain scale (sf&#8208;36 bps), and measure of intermittent and constant osteoarthritis pain (icoap).</a> Arthritis Care &amp; Research. 63 (11), 240-252 (2011).</li><li>Kehlet, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Acute+pain+control+and+accelerated+postoperative+surgical+recovery.">Acute pain control and accelerated postoperative surgical recovery.</a> Surgical Clinics. 79, 431-443 (1999).</li><li>McQuay, H., Moore, A., Justins, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Treating+acute+pain+in+hospital.">Treating acute pain in hospital.</a> British Medical Journal. 314 (7093), 1531-1535 (1997).</li><li>Brahnam, S., Chuang, C. F., Shih, F. Y., Slack, M. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=SVM+classification+of+neonatal+facial+images+of+pain.+Fuzzy+Logic+and+Applications+(revised+selected+papers+from+the+6th+International+Workshop,+WILF+2005,+Crema,+Italy,+September+15-17,+2005).">SVM classification of neonatal facial images of pain. Fuzzy Logic and Applications (revised selected papers from the 6th International Workshop, WILF 2005, Crema, Italy, September 15-17, 2005).</a> Lecture Notes in Computer Science. 3849, 121-128 (2005).</li><li>Basler, H. D., Bloem, R., Casser, H. R., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Ein+strukturiertes+Schmerzinterview+f&#252;r+geriatrische+Patienten.">Ein strukturiertes Schmerzinterview f&#252;r geriatrische Patienten.</a> Schmerz. 15, 164-171 (2001).</li><li>Zwakhalen, S. M. G., Hamers, J. P. H., Abu-Saad, H. H., Berger, M. P. F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Pain+in+elderly+people+with+severe+dementia:+a+systematic+review+of+behavioural+pain+assessment+tools.">Pain in elderly people with severe dementia: a systematic review of behavioural pain assessment tools.</a> BioMed Central Geriatrics. 6 (3), (2006).</li><li>Herr, K., Bjoro, K., Decker, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Tools+for+assessment+of+pain+in+nonverbal+older+adults+with+dementia:+a+state-of-the-science+review.">Tools for assessment of pain in nonverbal older adults with dementia: a state-of-the-science review.</a> Journal of Pain and Symptom Manage. 31 (2), 170-192 (2006).</li><li>Handel, E., Gnass, I. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Praxishandbuch ZOPA : Schmerzeinsch&#228;tzung bei Patienten mit kognitiven und/oder Bewusstseinsbeeintr&#228;chtigungen. , (2010).</li><li>Abbey, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+Abbey+pain+scale:+a+1-minute+numerical+indicator+for+people+with+end-stage+dementia.">The Abbey pain scale: a 1-minute numerical indicator for people with end-stage dementia.</a> International Journal Of Palliative Nursing. 10 (1), 6-13 (2004).</li><li>Gruss, S., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Pain+intensity+recognition+rates+via+biopotential+feature+patterns+with+support+vector+machines.">Pain intensity recognition rates via biopotential feature patterns with support vector machines.</a> PLoS ONE. 10, 1-14 (2015).</li><li>Walter, S., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automatic+pain+quantification+using+autonomic+parameters.">Automatic pain quantification using autonomic parameters.</a> Psychology &amp; Neuroscience. 7 (3), 363 (2014).</li><li>Werner, P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automatic+pain+assessment+with+facial+activity+descriptors.">Automatic pain assessment with facial activity descriptors.</a> IEEE Transactions on Affective Computing. 8 (3), 286-299 (2017).</li><li>Thiam, P., Kessler, V., Walter, S., Palm, G., Schwenker, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Audio-Visual+Recognition+of+Pain+Intensity.">Audio-Visual Recognition of Pain Intensity.</a> IAPR Workshop on Multimodal Pattern Recognition of Social Signals in Human-Computer Interaction. , (2016).</li><li>Niese, R., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Towards+Pain+Recognition+in+Post-Operative+Phases+Using+3D-based+Features+from+Video+and+Support+Vector+Machines.">Towards Pain Recognition in Post-Operative Phases Using 3D-based Features from Video and Support Vector Machines.</a> International Journal of Digital Content Technology and its Applications. 3 (4), 21-33 (2009).</li><li>Hammal, Z., Kunz, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Pain+monitoring:+A+dynamic+and+context-sensitive+system.">Pain monitoring: A dynamic and context-sensitive system.</a> Pattern Recognition. 45 (4), 1265-1280 (2012).</li><li>Werner, P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automatic+pain+recognition+from+video+and+biomedical+signals.">Automatic pain recognition from video and biomedical signals.</a> Pattern Recognition (ICPR), 22nd International Conference on Pattern Recognition. , 4582-4587 (2014).</li><li>Walter, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Diagnostik+der+Schmerzintensit&#228;t,+basierend+auf+multimodalen+Signalen+und+maschinellen+Lernen.">Diagnostik der Schmerzintensit&#228;t, basierend auf multimodalen Signalen und maschinellen Lernen.</a> Habilitationsschrift. , (2017).</li><li>Chu, Y., Zhao, X., Han, J., Su, Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Physiological+Signal-Based+Method+for+Measurement+of+Pain+Intensity.">Physiological Signal-Based Method for Measurement of Pain Intensity.</a> Frontiers in aneuroscience. 11, (2017).</li><li>K&#228;chele, M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Multimodal+data+fusion+for+person-independent,+continuous+estimation+of+pain+intensity.">Multimodal data fusion for person-independent, continuous estimation of pain intensity.</a> Engineering Applications of Neural Networks. , 275-285 (2015).</li><li>Walter, S., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+BioVid+Heat+Pain+Database+-+Data+for+the+advancement+and+systematic+validation+of+an+automated+pain+recognition+system.">The BioVid Heat Pain Database - Data for the advancement and systematic validation of an automated pain recognition system.</a> Cybernetics (CYBCONF), IEEE International Conference on Cybernetics. , 128-131 (2013).</li><li>Velana, M., Walter, S., Gruss, S., Werner, P., Al-Hamadi, A., Schwenker, F., Scherer, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+SenseEmotion+Database:+A+Multimodal+Database+for+the+Development+and+Systematic+Validation+of+an+Automatic+Pain-+and+Emotion-Recognition+System.">The SenseEmotion Database: A Multimodal Database for the Development and Systematic Validation of an Automatic Pain- and Emotion-Recognition System.</a> Multimodal Pattern Recognition of Social Signals in Human-Computer-Interaction. MPRSS 2016. Lecture Notes in Computer Science. , (2017).</li><li>Lautenbacher, S., Peters, J. H., Heesen, M., Scheel, J., Kunz, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Age+changes+in+pain+perception:+a+systematic-review+and+meta-analysis+of+age+effects+on+pain+and+tolerance+thresholds.">Age changes in pain perception: a systematic-review and meta-analysis of age effects on pain and tolerance thresholds.</a> Neuroscience &amp; Biobehavioral Reviews. 75, 104-113 (2017).</li><li>K&#225;llai, I., Barke, A., Voss, U. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+effects+of+experimenter+characteristics+on+pain+reports+in+women+and+men.">The effects of experimenter characteristics on pain reports in women and men.</a> Pain. 112 (1), 142-147 (2004).</li><li>McCambridge, J., Witton, J., Elbourne, D. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Systematic+review+of+the+Hawthorne+effect:+New+concepts+are+needed+to+study+research+participation+effects.">Systematic review of the Hawthorne effect: New concepts are needed to study research participation effects.</a> Journal of Clinical Epidemiology. 67 (3), 267-277 (2014).</li></ol>

The authors have nothing to disclose.

The presented protocol focuses on the experimental elicitation of thermal (heat) and electrical pain while recording physiological, visual and paralinguistic signals. This novel approach, combining two pain models with different stimuli intensities and two different stimuli durations (phasic and tonic), offers a broad perspective about the psychophysiological patterns and expressions of pain. However, for the realization of this protocol several steps need to be considered.&#160;
In general, if working with pain stimuli it is crucial to ensure the safety of the subjects. All pain stimuli have to be highly controlled and should only be carried out by experienced experimenters.&#160; 
Furthermore, for recording and collecting reliable and high quality data, the proper attachment of devices (electrodes), the perfect functioning of recording devices and a smooth communication between computers is highly recommended. All sources of interferences should be eliminated or reduced to a minimum. To guarantee consistency between participants, it is important to provide standardized instructions and unvarying experimental conditions.&#160;
According to our experience, finding suitable participants who meet all criteria and are willing to receive numerous painful stimuli, takes a long time and is quite challenging. In addition to that, the monetary compensation has to be high enough to attract subjects to the study. Especially persons between 30 and 50 years are hard to find. This may be because the experiment is too long (ca. 4 hours, including arrival and departure) and they have to take half a day off from work.
Because the safety of the participants is of top priority, pain induction may need to be restricted. Due to ethical guidelines, the stimulus intensities must not exceed certain levels to prevent burns and unconsciousness in terms of thermal and electrical pain induction, respectively. A general cutoff of intensities may result in a ceiling effect as some subjects may reach the intensity limits before feeling intolerable pain. In this study, approximately 42% (considering&#160;thermal calibration part 1 and 2) of the participants reached the thermal cutoffs (see representative results). As they did not reach their &#8220;real&#8221; pain tolerances, their physiological responses to the highest thermal stimuli might behave differently in contrast to physiological responses of subjects who reached them. If so, mixing these two groups could influence classification results in terms of pain recognition.
An important point to address is the pain modalities in this experiment. Participants are subjected only to thermal and electrical pain stimuli (due to the fact that these are highly controllable in an experimental setting). Thus, if examining pain patterns regarding quality, findings may not translate to other pain modalities such as pressure, chemical or visceral pain. 
The same consideration on transferability of results applies to the study sample. The protocol is ethically restricted to healthy adults. For example, it does not include children or cognitively and verbally impaired persons. Furthermore, in our study only European people participated. Also here, analytical results may not apply to groups not considered in this experiment.
Another limitation may concern the Hawthorne effect24: The subjects are aware that they are being filmed/observed in the study. This might change their behavior.
Compared to existing pain databases, the protocol provides significant advantages for analyzing pain response patterns as it combines two pain models and two time courses (phasic and tonic): Besides the intensity and duration of pain, it also considers the quality of pain. As thermal pain is described differently than electrical pain (e.g., burning vs. sharp), it may also differ in the pain reactions. If so, those findings could link a pain response pattern to the underlying source of pain. Furthermore, the study is multi-modal to widen the range of pain investigation opportunities: Employing 5 psychophysiological signals, 2 face (front/side) camera signals, 1 body view camera signal, 1 thermal camera and 1 audio signal, pain may be analyzed and assessed more precisely.&#160;
For a more complex investigation of pain response patterns, future extensions of this method should include more biosignals such as electroencephalography (EEG), body temperature and respiration. It would also be of great benefit to employ controlled pressure as a further pain model. Researchers aiming at automatic pain recognition via data gathered with this protocol should further test promising machine learning models with clinical control groups.&#160;

Pain is a very personal and unpleasant sensation that is perceived differently by everyone. It lasts from seconds to months and may vary in its quality (throbbing, sharp, burning, etc.). If treated inadequately, pain influences physical and psychological functions of the body, reduces the quality of life, and bears the risk of becoming a chronic condition. In clinical care, the accurate assessment of pain intensity and quality is highly relevant to provide successful pain management1,2. Gold standard methods for assessing pain, such as the visual analog scales (VAS), the numeric rating scale (NRS), or the McGill Pain Questionnaire3, rely on self-reports of patients and, thus, only work sufficiently with cognitively and verbally unimpaired persons. Consequently, all those established methods lack validity and reliability when it comes to neonates4, delirious, somnolent, sedated, or ventilated patients5, or people suffering from dementia6,7. In addition to or as an alternative to self-report scales, methods to measure pain through observation by trained personnel (e.g., the Zurich Observation Pain Assessment8 or the Abbey Pain Scale9) have been developed in recent years. Nevertheless, even these tools suffer from limitations in reliability and validity, as even trained raters cannot guarantee an objective assessment. Furthermore, the application is often too time-consuming for clinical staff when pain assessment should be done on a regular basis.&#160;
Several research teams have focused on developing automated pain recognizing systems, which allow for measuring pain by means of physiological, visual, and/or paralinguistic signal sets as new approaches for assessing and monitoring pain and its intensities objectively. Previous studies show promising results in detecting and differentiating pain10,11,12,13,16,17,18 or discriminating pain from basic emotions14,15 based solely on one of the signal sets10,11,12,13,14,15 as well as on a combination/fusion16,17,19 of the sets. The abovementioned modalities react almost autonomously to stressful stimuli such as pain. Using them has the advantage that they do not require a person&#8217;s ability to report her/his pain. Such individuals would greatly benefit from an objective pain recognition system which incorporates such modalities. Data sets consisting of reactions to elicited pain provide precious information for analyzing pain patterns and developing practical applications for detecting and monitoring pain. Amongst others, Walter et al.20 created the &#8220;BioVid Heat Pain Database&#8221;, a multimodal database that is publicly available and provides data from short-time induced painful heat stimuli and corresponding psychophysiological and visual reactions. The &#8220;SenseEmotion Database&#8221; of Velana et al.21 includes biosignals, videos, and paralinguistic information from volunteers affected by phasic heat pain and emotional stimuli.
While these databases are well suited for examining pain reactions, they are mostly based on one specific pain model. As pain differs in its quality (supposedly depending on the pain model) and in its duration, it also may differ in its physiological, visual, and paralinguistic correlates. To the best of the authors&#8217; knowledge, no multimodal studies or databases exist that combine two or more pain models and vary pain stimuli in intensity and duration in order to not only detect pain patterns but also distinguish between pain qualities.
This paper provides a protocol on how to conduct a complex psychophysiological experiment to elicit pain and simultaneously record physiological responses (e.g., ECG, EMG of Musculus trapezius, corrugator supercilii, and zygomaticus major, SCL) as well as video (e.g., facial expressions, body gestures, facial skin temperature) and audio data. Participants are stimulated with short (phasic) and longer lasting (tonic) heat and electrical pain stimuli that differ in intensity. A calibration phase prior to the experiment determines pain thresholds for each subject individually.
The study aims at collecting multimodal data for investigating pain (patterns) regarding intensity, quality and length by means of statistical methods, machine learning algorithms, etc. Additionally, the already collected data is planned to be published for academic research purposes under the name &#8220;X-ITE (Experimentally Induced Thermal and Electrical) Pain Database&#8221;. It may extend existing databases, such as BioVid Heat Pain and SenseEmotion20,21, and contribute to the further development, improvement, and/or evaluation of automated pain recognition systems in matters of validity, reliability, and real-time recognition.&#160;
The rest of the paper is organized in the following way. The protocol describes how to carry out the pain elicitation study step-by-step. Then, the representative results present the outcome of the experiment. Finally, the discussion covers critical steps, limitations, and benefits of the study followed by suggestions for future extensions.

<table border="0" cellpadding="0" cellspacing="0" style="width:1531px;" width="1532">
	<tbody>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">PATHWAY Model ATS</td>
			<td style="width:514px;">Medoc Ltd., Ramat Yishai, Israel</td>
			<td style="width:178px;"></td>
			<td style="width:422px;">Thermal Stimulator</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">30 mm x 30 mm ATS Thermode</td>
			<td style="width:514px;">Medoc Ltd., Ramat Yishai, Israel</td>
			<td style="width:178px;"></td>
			<td style="width:422px;">Thermode</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">PATHWAY Software Arbel 6.3.7.22.1</td>
			<td style="width:514px;">Medoc Ltd., Ramat Yishai, Israel</td>
			<td style="width:178px;"></td>
			<td>Thermal Stimulator Software</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">Digitimer DS7A Current Stimulator</td>
			<td style="width:514px;">Digitimer Ltd., Hertfordshire, UK</td>
			<td style="width:178px;"></td>
			<td style="width:422px;">Electrical Stimulator</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">Inquisit 5</td>
			<td style="width:514px;">Millisecond Software, Seattle, WA, USA</td>
			<td style="width:178px;"></td>
			<td style="width:422px;">Software for triggering electrical stimuli</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">Analogue-To-Digital Converter</td>
			<td style="width:514px;">Wissenschaftliche Werkstatt Elektronik, University of Ulm, Ulm, Germany</td>
			<td style="width:178px;">custom built</td>
			<td></td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">BIOPAC MP150 System</td>
			<td style="width:514px;">BIOPAC Systems, Inc., Goleta, CA, USA</td>
			<td style="width:178px;"></td>
			<td>Biosignal Recording Hardware</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;">AcqKnowledge Software 4.1.1</td>
			<td style="width:514px;">BIOPAC Systems, Inc., Goleta, CA, USA</td>
			<td style="width:178px;"></td>
			<td>Biosignal Recording Software</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">NTG-2 Dual Powered Directional Condenser Microphone</td>
			<td style="width:514px;">R&#216;DE Microphones, Silverwater, Australia</td>
			<td style="width:178px;"></td>
			<td>Audio Recording Microphone</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">Kinect v2</td>
			<td style="width:514px;">Microsoft, Redmond, WA, USA</td>
			<td style="width:178px;"></td>
			<td>Body View Camera</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">AV Pike F-145C</td>
			<td style="width:514px;">Allied Vision Technologies GmbH, Stadtroda, Germany</td>
			<td style="width:178px;"></td>
			<td>Face Camera (frontal view)</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">AV Prosilica GT 1600C</td>
			<td style="width:514px;">Allied Vision Technologies GmbH, Stadtroda, Germany</td>
			<td style="width:178px;"></td>
			<td>Face Camera (side view)</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">PIR uc 180 Thermal Camera</td>
			<td style="width:514px;">InfraTec GmbH, Dresden, Germany</td>
			<td style="width:178px;"></td>
			<td>Thermal Face Camera</td>
		</tr>
		<tr height="42">
			<td height="42" style="height:42px;width:419px;">Synchronization Hardware</td>
			<td style="width:514px;">Werkstatt, IIKT, University of Magdeburg, Magdeburg, Germany</td>
			<td style="width:178px;">custom built</td>
			<td style="width:422px;">Hardware triggering of cameras, trigger signal is recorded by BIOPAC and Audacity</td>
		</tr>
		<tr height="42">
			<td height="42" style="height:42px;width:419px;">Recording and Synchronization Software</td>
			<td style="width:514px;">Philipp Werner, Neuro-Information Technology, University of Magdeburg, Magdeburg, Germany</td>
			<td style="width:178px;">custom software</td>
			<td style="width:422px;">Real-time recording, offline video encoding, and offline synchronization</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">Examination Couch</td>
			<td style="width:514px;">ClinicalCare GmbH, Bremen, Germany</td>
			<td style="width:178px;"></td>
			<td style="width:422px;"></td>
		</tr>
		<tr height="42">
			<td height="42" style="height:42px;width:419px;">Ag-AgCl Electrodes EL254 / EL254S (Reusable, 4mm recording diameter)</td>
			<td style="width:514px;">BIOPAC Systems, Inc., Goleta, CA, USA</td>
			<td style="width:178px;"></td>
			<td>Used to record EMG M. corrugator and M. zygomaticus</td>
		</tr>
		<tr height="44">
			<td height="44" style="height:44px;width:419px;">Ag-AgCl Electrodes BlueSensor P (Disposable, skin contact size: 34 mm diameter, measuring area 154 mm2)</td>
			<td style="width:514px;">Ambu GmbH, Bad Nauheim, Germany</td>
			<td style="width:178px;"></td>
			<td>Used to record ECG and EMG M. trapezius. Also used for electrical stimulation</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">Audacity 2.1.2</td>
			<td style="width:514px;">Dominic Mazzoni (Audacity)</td>
			<td style="width:178px;"></td>
			<td>Audio Recording Software</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">Cold Gel Pack</td>
			<td style="width:514px;">C+V Pharma Depot GmbH, Versmold, Germany</td>
			<td style="width:178px;"></td>
			<td></td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">Panthenol 50mg/g</td>
			<td style="width:514px;">ratiopharm GmbH, Ulm, Germany</td>
			<td style="width:178px;"></td>
			<td>Ointment</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">Alumnium Profiles</td>
			<td style="width:514px;">item Industrietechnik GmbH, Solingen, Germany</td>
			<td style="width:178px;"></td>
			<td>Used to install all cameras and microphone</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">Electrode Gel GEL1</td>
			<td style="width:514px;">BIOPAC Systems, Inc., Goleta, CA, USA</td>
			<td style="width:178px;"></td>
			<td></td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:419px;">ELPREP Skin Preparation Gel</td>
			<td style="width:514px;">BIOPAC Systems, Inc., Goleta, CA, USA</td>
			<td style="width:178px;"></td>
			<td></td>
		</tr>
	</tbody>
</table>

multi-modal signals for analyzing pain responses to thermal and electrical stimuli

The assessment of pain relies mostly on methods that require a person to communicate. However, for people with cognitive and verbal impairments, existing methods are not sufficient as they lack reliability and validity. To approach this problem, recent research focuses on an objective pain assessment facilitated by parameters of responses derived from physiology, and video and audio signals. To develop reliable automated pain recognition systems, efforts have been made in creating multimodal databases in order to analyze pain and detect valid pain patterns. While the results are promising, they only focus on discriminating pain or pain intensities versus no pain. In order to advance this, research should also consider the quality and duration of pain as they provide additional valuable information for more advanced pain management. To complement existing databases and the analysis of pain regarding quality and length, this paper proposes a psychophysiological experiment to elicit, measure, and collect valid pain reactions. Participants are subjected to painful stimuli that differ in intensity (low, medium, and high), duration (5 s / 1 min), and modality (heat / electric pain) while audio, video (e.g., facial expressions, body gestures, facial skin temperature), and physiological signals (e.g., electrocardiogram [ECG], skin conductance level [SCL], facial electromyography [EMG], and EMG of M. trapezius) are being recorded. The study consists of a calibration phase to determine a subject&#8217;s individual pain range (from low to intolerable pain) and a stimulation phase in which pain stimuli, depending on the calibrated range, are applied. The obtained data may allow refining, improving, and evaluating automated recognition systems in terms of an objective pain assessment. For further development of such systems and to investigate pain reactions in more detail, additional pain modalities such as pressure, chemical, or cold pain should be included in future studies. Recorded data of this study will be released as the &#8220;X-ITE Pain Database&#8221;.

Pain is perceived differently by any person and may express itself diversely in facial expressions, paralinguistic and/or physiological signals. The design of this study is suitable to analyze pain responses in numerous ways with respect to the underlying aims. The obtained data may allow answering research questions, such as: Are there specific pain response patterns? Do they differ regarding pain model and duration?
A total of 134 subjects participated in our experiment. The sex ratio was 50/50. We divided them in the following age groups: 1) 18-29 years (N = 49, 23 men, 26 women), 2) 30-39 years (N = 45, 23 men, 22 women), 3) 40-50 years (N = 40, 21 men, 19 women). The average age of all subjects was 31.4 (SD = 9.7), of all men = 33.4 (SD = 9.3) and of all women = 32.9 (SD = 10.2) years. The study took place at the Department of Medical Psychology of the University of Ulm, Germany.
The main outcome of this protocol is a data set of audio, video and psychophysiological signals reflecting the subjects&#8217; responses to pain stimuli. Table 1 provides a general overview on the technical features of the recorded signals and on the numbers of induced pain stimuli in the study.&#160;
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td colspan="5">Technical Features</td>
		</tr>
		<tr>
			<td>Signal:</td>
			<td colspan="2">Sampling Rate:</td>
			<td colspan="2">Attributes:</td>
		</tr>
		<tr>
			<td>Audio</td>
			<td colspan="2">44100 Hz</td>
			<td colspan="2">Mono, MP3 320 kbps</td>
		</tr>
		<tr>
			<td>Camera 1 (face, frontal view)</td>
			<td colspan="2">25 Hz</td>
			<td colspan="2">Color video: resolution 1384 x 1032, 
			HEVC encoded with libx265 (CRF 16, preset medium)</td>
		</tr>
		<tr>
			<td>Camera 2 (face, side view)</td>
			<td colspan="2">25 Hz</td>
			<td colspan="2">Color video: resolution 1620 x 840, 
			HEVC encoded with libx265 (CRF 16, preset medium)</td>
		</tr>
		<tr>
			<td>Body Camera</td>
			<td colspan="2">ca. 30 Hz</td>
			<td colspan="2">Color video: resolution 1500 x 600, 
			HEVC encoded with libx265 (CRF 16, preset medium); 
			Depth video: resolution 500 x 200, lossless encoding</td>
		</tr>
		<tr>
			<td>Thermal Camera</td>
			<td colspan="2">ca. 120.8 Hz</td>
			<td colspan="2">Surface temperature video: resolution 120 x 160, 
			grayscale MPEG-4-AVC encoded with&#160; libx264 
			(CRF 0,&#160; preset veryfast), 
			encoded temperature range 26.5-52.0 &#176;C (steps of 0.1)</td>
		</tr>
		<tr>
			<td>ECG</td>
			<td colspan="2">1000 Hz</td>
			<td colspan="2">Hardware filtered via BioPac: 35 Hz LP, 0.5 Hz HP, 
			50 Hz notch filter</td>
		</tr>
		<tr>
			<td>SCL</td>
			<td colspan="2">1000 Hz</td>
			<td colspan="2">Hardware filtered via BioPac: 10 Hz LP, no HP, 
			no notch filter</td>
		</tr>
		<tr>
			<td>EMG M. trapezius</td>
			<td colspan="2">1000 Hz</td>
			<td colspan="2">Hardware filtered via BioPac: 500 Hz LP, 10 Hz HP, 
			no notch filter</td>
		</tr>
		<tr>
			<td>EMG M. corrugator supercilii</td>
			<td colspan="2">1000 Hz</td>
			<td colspan="2">Hardware filtered via BioPac: 500 Hz LP, 10 Hz HP, 
			no notch filter</td>
		</tr>
		<tr>
			<td>EMG M. zygomaticus major</td>
			<td colspan="2">1000 Hz</td>
			<td colspan="2">Hardware filtered via BioPac: 500 Hz LP, 10 Hz HP, 
			no notch filter</td>
		</tr>
		<tr>
			<td>Stimuli</td>
			<td colspan="2">Thermal</td>
			<td colspan="2">Electrical</td>
		</tr>
		<tr>
			<td>Subjects:</td>
			<td>Phasic Stimuli (5 s):</td>
			<td>Tonic Stimuli (60 s):</td>
			<td>Phasic Stimuli (5 s):</td>
			<td>Tonic Stimuli (60 s):</td>
		</tr>
		<tr>
			<td>Per subject</td>
			<td>90 (30 per intensity)</td>
			<td>3 (1 per intensity)</td>
			<td>90 (30 per intensity)</td>
			<td>3 (1 per intensity)</td>
		</tr>
		<tr>
			<td>All (N = 134)</td>
			<td>12060 (4020 per intensity)</td>
			<td>402 (134 per intensity)</td>
			<td>12060 (4020 per intensity)</td>
			<td>402 (134 per intensity)</td>
		</tr>
		<tr>
			<td>Men (n = 67)</td>
			<td>6030 (2010 per intensity)</td>
			<td>201 (67 per intensity)</td>
			<td>6030 (2010 per intensity)</td>
			<td>201 (67 per intensity)</td>
		</tr>
		<tr>
			<td>Women (n = 67)</td>
			<td>6030 (2010 per intensity)</td>
			<td>201 (67 per intensity)</td>
			<td>6030 (2010 per intensity)</td>
			<td>201 (67 per intensity)</td>
		</tr>
	</tbody>
</table>
Table 1: Technical features and number of induced stimuli. The upper half (Technical Features) shows the sampling rates and attributes of the specific signals. The lower half (Stimuli) shows the numbers of the induced specific (thermal/electrical) pain stimuli for one subject, for all subjects and for each gender. (MP3 = Moving Picture Experts Group Layer-3 Audio, kbps = kilobits per second, HEVC = High Efficiency Video Coding, CRF = Constant Rate Factor, MPEG-4-AVC = Motion Picture Experts Group Layer-4 Video Advanced Video Coding, Hz = Hertz, &#176;C = degrees Celsius, s = seconds, ECG = Electrocardiogram, SCL = Skin Conductance Level, EMG = Electromyography, LP = low-pass filter, HP = high-pass filter, M. = Musculus).
A secondary outcome concerning the calibration phase of the study is presented in Table 2. It shows the mean stimulation temperatures and currents of pain intensities 1 and 3 (as calculated in step 5.11 of the protocol) for all subjects and additionally for the male and female subgroup.&#160;
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>Stimuli</td>
			<td colspan="4">Thermal [in &#176;C] mean (SD)</td>
			<td colspan="4">Electrical [in mA] mean (SD)</td>
		</tr>
		<tr>
			<td>Subjects</td>
			<td>pH1</td>
			<td>pH3</td>
			<td>tH1</td>
			<td>tH3</td>
			<td>pE1</td>
			<td>pE3</td>
			<td>tE1</td>
			<td>tE3</td>
		</tr>
		<tr>
			<td>All (N = 134)</td>
			<td>44.03 (2.25)</td>
			<td>49.17 (1.20)</td>
			<td>42.50 (2.14)</td>
			<td>47.76 (1.02)</td>
			<td>1.63 (0.94)</td>
			<td>5.64 (2.72)</td>
			<td>1.69 (1.12)</td>
			<td>5.70 (2.59)</td>
		</tr>
		<tr>
			<td>Men (n = 67)</td>
			<td>44.56 (2.18)</td>
			<td>49.48 (0.89)</td>
			<td>43.11 (1.98)</td>
			<td>47.93 (1.04)</td>
			<td>1.94 (1.01)</td>
			<td>6.83 (3.02)</td>
			<td>1.96 (1.16)</td>
			<td>6.90 (2.72)</td>
		</tr>
		<tr>
			<td>Women (n = 67)</td>
			<td>43.51 (2.74)</td>
			<td>48.87 (1.39)</td>
			<td>41.89 (2.14)</td>
			<td>47.59 (0.98)</td>
			<td>1.32 (0.75)</td>
			<td>4.45 (1.70)</td>
			<td>1.43 (1.01)</td>
			<td>4.51 (1.80)</td>
		</tr>
	</tbody>
</table>
Table 2: Mean stimulation temperatures and currents of pain intensities 1 and 3. (pH1, pH3 = phasic heat pain with intensity 1, 3; tH1, tH3 = tonic heat pain with intensity 1, 3; pE1, pE3 = phasic electrical pain with intensity 1, 3; tE1, tE3 = tonic electrical pain with intensity 1, 3; &#176;C = degrees Celsius; mA = milliampere, SD = standard deviation).
If all steps of the protocol are conducted carefully and no technical problems occur (in terms of computer or recording device crashes, etc.), a successful outcome may look similar as depicted in Figure 4. All signals are of high quality and not affected by external sources of interferences. The participant is clearly visible in every camera.&#160;
<img alt="Figure 4" class="xfigimg" src="/files/ftp_upload/59057/59057fig4.jpg" /> 
Figure 4: Example data from a successful experiment. The figure depicts recorded signals a few seconds before, during and after an intense pain stimulus. All signals are non-filtered and synchronized in time. For clarity, only representative screenshots of the video signals are shown here. (EMG = Electromyography, SCL = Skin Conductance Level, ECG = Electrocardiogram, M. = Musculus, s = seconds). <a href="https://www.jove.com/files/ftp_upload/59057/59057fig4large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
However, unexpected incidents may cause the data to become noisy or corrupted. Besides computer or recording device crashes, the coming-off of electrodes (especially reusable electrodes with small diameter which are attached by means of double-sided adhesive collars) mostly leads to unusable signals. As an example for a sub-optimal data set, Figure 5 shows the moment when an EMG electrode comes off and renders the corresponding signal useless.&#160;
<img alt="Figure 5" class="xfigimg" src="/files/ftp_upload/59057/59057fig5.jpg" /> 
Figure 5: Example data from a sub-optimal experiment. The red circle indicates the time one of the EMG electrodes (M. zygomaticus major) fell off the subject&#8217;s cheek. This might have been due to sweat or head movement. From this moment on, the signal was lost. (EMG = Electromyography, SCL = Skin Conductance Level, ECG = Electrocardiogram, M. = Musculus, s = seconds). <a href="https://www.jove.com/files/ftp_upload/59057/59057fig5large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
Due to ethical guidelines, the maximum intensities of thermal and electrical stimuli had to be restricted. Regarding thermal calibration control part (see supplementary file), 37 subjects (31 men, 6 women) reached the given cutoff of 50.5 &#176;C (ratio = 37/134 = 27.61 %). As for thermal calibration part 1, 60 participants (39 men, 21 women) reached the cutoff of 50.0 &#176;C (ratio = 60/134 = 44.78 %) and concerning part 2, 57 persons (37 men, 20 women) reached the cutoff of 49.5 &#176;C (ratio = 57/134 = 42.54 %). The cutoff for both electrical calibration parts was 25 mA. None of the 134 subjects reached it.
As we plan to publish the data (see next paragraph), the data sets of participants who have reached the cutoffs will additionally be marked and their subjective pain ratings for the corresponding cutoffs will be included.
We would like to point out that the main focus of the protocol is obtaining multi-modal signals for analyzing thermal and electrical pain. Therefore, no other results are discussed here. After checking and excluding data sets due to missing data or rejected written consent for data sharing, the data sets of this study will be made available under the name &#8220;X-ITE Pain Database&#8221;. For further information on when and how to obtain the X-ITE Pain DB please visit https://github.com/philippwerner/pain-database-list.
Supplementary File 1.&#160;<a href="https://www.jove.com/files/ftp_upload/59057/Supplementary_File_1.docx" target="_blank">Please click here to download this file.</a>

Watch this Scientific Journal Video about Multi-Modal Signals for Analyzing Pain Responses to Thermal and Electrical Stimuli Video at JoVE.com

Multi-Modal Signals for Analyzing Pain Responses to Thermal and Electrical Stimuli Video (Video) | JoVE

This article focuses on the experimental elicitation of pain through heat (thermal) and electrical stimulation while recording physiological, visual, and paralinguistic responses. It aims at collecting valid multimodal data for analyzing pain based on its intensity, quality, and duration.&#160;

Multi-Modal Signals for Analyzing Pain Responses to Thermal and Electrical Stimuli

The assessment of pain relies mostly on methods that require a person to communicate. However, for people with cognitive and verbal impairments, existing ...

Research

JoVE Journal

Behavior

Use of the Operant Orofacial Pain Assessment Device (OPAD) to Measure Changes in Nociceptive Behavior

Use of the Operant Orofacial Pain Assessment Device OPAD to Measure Changes in Nociceptive Behavior Video (Video) | JoVE

We present an operant  system for the detection of pain in awake, conscious rodents. The Orofacial  Pain Assessment Device (OPAD) assesses pain behaviors ...

Simultaneous Scalp Electroencephalography (EEG), Electromyography (EMG), and Whole-body Segmental Inertial Recording for Multi-modal Neural Decoding

Simultaneous Scalp Electroencephalography EEG, Electromyography EMG, and Whole-body Segmental Inertial Recording for Multi-modal Neural Decoding Video (Video) | JoVE

Recent  studies support the involvement of supraspinal networks in control of bipedal  human walking. Part of this evidence  encompasses studies, ...

Contextual and Cued Fear Conditioning Test Using a Video Analyzing System in Mice

Contextual and Cued Fear Conditioning Test Using a Video Analyzing System in Mice Video (Video) | JoVE

The contextual and cued fear conditioning test is one of the behavioral tests that assesses the ability of mice to learn and remember an association ...

Early Metamorphic Insertion Technology for Insect Flight Behavior Monitoring

Early Metamorphic Insertion Technology for Insect Flight Behavior Monitoring Video (Video) | JoVE

Early Metamorphosis Insertion Technology (EMIT) is a novel methodology for integrating microfabricated neuromuscular recording and actuation platforms on ...

Irrelevant Stimuli and Action Control: Analyzing the Influence of Ignored Stimuli via the Distractor-Response Binding Paradigm

Irrelevant Stimuli and Action Control: Analyzing the Influence of Ignored Stimuli via the Distractor-Response Binding Paradigm Video (Video) | JoVE

Selection tasks in which simple stimuli (e.g. letters) are presented and a target stimulus has to be selected against one or more distractor stimuli are ...

Determining Pain Detection and Tolerance Thresholds Using an Integrated, Multi-Modal Pain Task Battery

Determining Pain Detection and Tolerance Thresholds Using an Integrated, Multi-Modal Pain Task Battery Video (Video) | JoVE

Human pain models are useful in the assessing the analgesic effect of drugs, providing information about a drug&#39;s pharmacology and identify ...

A Protocol of Manual Tests to Measure Sensation and Pain in Humans

A Protocol of Manual Tests to Measure Sensation and Pain in Humans Video (Video) | JoVE

Numerous qualitative and quantitative techniques can be used to test sensory nerves and pain in both research and clinical settings. The current study ...

A Method for Manipulating Blood Glucose and Measuring Resulting Changes in Cognitive Accessibility of Target Stimuli

A Method for Manipulating Blood Glucose and Measuring Resulting Changes in Cognitive Accessibility of Target Stimuli Video (Video) | JoVE

Much research in social psychology has investigated the impact of bodily energy need on cognition and decision-making.  As such, blood glucose, the body's ...

Simultaneous Recordings of Cortical Local Field Potentials and Electrocorticograms in Response to Nociceptive Laser Stimuli from Freely Moving Rats

Simultaneous Recordings of Cortical Local Field Potentials and Electrocorticograms in Response to Nociceptive Laser Stimuli from Freely Moving Rats Video (Video) | JoVE

Electrocortical responses, elicited by laser heat pulses that selectively activate nociceptive free nerve endings, are widely used in many animal and ...

Generating Strictly Controlled Stimuli for Figure Recognition Experiments

Generating Strictly Controlled Stimuli for Figure Recognition Experiments Video (Video) | JoVE

This protocol introduces a method for generating strictly controlled and objectively defined stimuli for figure recognition experiments. A (6, n) figure ...