Name: protocol for data collection and analysis applied to automated facial expression analysis technology and temporal analysis for sensory evaluation
Uploaded: 2016-08-26T13:00:00
Description: Watch this Scientific Journal Video about Protocol for Data Collection and Analysis Applied to Automated Facial Expression Analysis Technology and Temporal Analysis for Sensory Evaluation at JoVE.com

This project was funded, in part, by ConAgra Foods (Omaha, NE, USA), the Virginia Agricultural Experiment Station, the Hatch Program of the National Institute of Food and Agriculture, U.S. Department of Agriculture, and the Virginia Tech Water INTERface Interdisciplinary Graduate Education Program.

Ethics Statement: This study was pre-approved by Virginia Tech Institutional Review Board (IRB) (IRB 14-229) prior to starting the project.
Caution: Human subject research requires informed consent prior to participation. In addition to IRB approval, consent for use of still or video images is also required prior to releasing any images for print, video, or graphic imaging. Additionally, food allergens are disclosed prior to testing. Participants are asked prior to panel start if they have any intolerance, allergies or other concerns....

<ol>
	<li>De Wijk, R. A., Kooijman, V., Verhoeven, R. H. G., Holthuysen, N. T. E., De Graaf, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Autonomic+nervous+system+responses+on+and+facial+expressions+to+the+sight,+smell,+and+taste+of+liked+and+disliked+foods.">Autonomic nervous system responses on and facial expressions to the sight, smell, and taste of liked and disliked foods.</a> Food Qual Prefer. 26 (2), 196-203 (2012).</li><li>De Wijk, R. A., He, W., Mensink, M. G. J., Verhoeven, R. H. G., De Graaf, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=ANS+responses+and+facial+expression+differentiate+between+the+taste+of+commercial+breakfast+drinks.">ANS responses and facial expression differentiate between the taste of commercial breakfast drinks.</a> PLoS ONE. 9 (4), 1-9 (2014).</li><li>He, W., Boesveldt, S., De Graaf, C., De Wijk, R. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Behavioural+and+physiological+responses+to+two+food+odours.">Behavioural and physiological responses to two food odours.</a> Appetite. 59 (2), 628 (2012).</li><li>He, W., Boesveldt, S., De Graaf, C., De Wijk, R. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Dynamics+of+autonomic+nervous+system+responses+and+facial+expressions+to+odors.">Dynamics of autonomic nervous system responses and facial expressions to odors.</a> Front Psychol. 5 (110), 1-8 (2014).</li><li>Danner, L., Sidorkina, L., Joechl, M., Duerrschmid, K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Make+a+face!+Implicit+and+explicit+measurement+of+facial+expressions+elicited+by+orange+juices+using+face+reading+technology.">Make a face! Implicit and explicit measurement of facial expressions elicited by orange juices using face reading technology.</a> Food Qual Prefer. 32 (2014), 167-172 (2013).</li><li>Danner, L., Haindl, S., Joechl, M., Duerrschmid, K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Facial+expression+and+autonomous+nervous+system+responses+elicited+by+tasting+different+juices.">Facial expression and autonomous nervous system responses elicited by tasting different juices.</a> Food Res Int. 64 (2014), 81-90 (2014).</li><li>Arnade, E. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Measuring consumer emotional response to tastes and foods through facial expression analysis [thesis]. , 1-187 (2013).</li><li>Leitch, K. A., Duncan, S. E., O'Keefe, S., Rudd, R., Gallagher, D. L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Characterizing+consumer+emotional+response+to+sweeteners+using+an+emotion+terminology+questionnaire+and+facial+expression+analysis.">Characterizing consumer emotional response to sweeteners using an emotion terminology questionnaire and facial expression analysis.</a> Food Res Int. 76, 283-292 (2015).</li><li>Crist, C. A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Application+of+emotional+facial+analysis+technology+to+consumer+acceptability+using+a+basic+tastes+model.">Application of emotional facial analysis technology to consumer acceptability using a basic tastes model.</a> , (2014).</li><li>Garcia-Burgos, D., Zamora, M. C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Facial+affective+reactions+to+bitter-tasting+foods+and+body+mass+index+in+adults.">Facial affective reactions to bitter-tasting foods and body mass index in adults.</a> Appetite. 71 (2013), 178-186 (2013).</li><li>Garcia-Burgos, D., Zamora, M. C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Exploring+the+hedonic+and+incentive+properties+in+preferences+for+bitter+foods+via+self-reports,+facial+expressions+and+instrumental+behaviours.">Exploring the hedonic and incentive properties in preferences for bitter foods via self-reports, facial expressions and instrumental behaviours.</a> Food Qual Prefer. 39 (2015), 73-81 (2015).</li><li>Lewinski, P., Fransen, M. L., Tan, E. S. H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Predicting+advertising+effectiveness+by+facial+expressions+in+response+to+amusing+persuasive+stimuli.">Predicting advertising effectiveness by facial expressions in response to amusing persuasive stimuli.</a> J. Neurosci. Psychol. Econ. 7 (1), 1-14 (2014).</li><li>Ekman, P., Friesen, W. V. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Facial+action+coding+system:+A+technique+for+the+measurement+of+facial+movement.">Facial action coding system: A technique for the measurement of facial movement.</a> , (1978).</li><li>Viola, P., Jones, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Rapid+object+detection+using+a+boosted+cascade+of+simple+features.">Rapid object detection using a boosted cascade of simple features.</a> Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern. 1, 511-518 (2001).</li><li>Sung, K. K., Poggio, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Example-based+learning+for+view-based+human+face+detection.">Example-based learning for view-based human face detection.</a> IEEE Trans. Pattern Anal. Mach. Intell. 20 (1), 39-51 (1998).</li><li>. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> FaceReader 5&#8482; Technical Specifications. , (2014).</li><li>Cootes, T., Taylor, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Statistical models of appearance for computer vision: Technical report. , (2000).</li><li>Bishop, C. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Neural networks for pattern recognition. , (1995).</li><li>Lewinski, P., den Uyl, T. M., Butler, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automated+facial+coding:+validation+of+basic+emotions+and+FACS+AUs+in+FaceReader.">Automated facial coding: validation of basic emotions and FACS AUs in FaceReader.</a> J. Neurosci. Psychol. Econ. 7 (4), 227-236 (2014).</li><li>Noldus Information Technology. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> FaceReader Reference Manual Version 6. , (2014).</li><li>Alves, N. T., Fukusima, S. S., Aznar-Casanova, J. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Models+of+brain+asymmetry+in+emotional+processing.">Models of brain asymmetry in emotional processing.</a> Psychol Neurosci. 1 (1), 63-66 (2008).</li><li>Costello, M., Clark, S., Clark, S., Costello, M., Drake, M., Bodyfelt, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Preparation+of+samples+for+instructing+students+and+staff+in+dairy+products+evaluation+(Appendix+F).">Preparation of samples for instructing students and staff in dairy products evaluation (Appendix F).</a> The sensory evaluation of dairy foods. , 551-560 (2009).</li><li>Porcherot, C., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=How+do+you+feel+when+you+smell+this?+Optimization+of+a+verbal+measurement+of+odor-elicited+emotions.">How do you feel when you smell this? Optimization of a verbal measurement of odor-elicited emotions.</a> Food Qual Prefer. 21, 938-947 (2010).</li><li>Warrenburg, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Effects+of+fragrance+on+emotions:+Moods+and+physiology.">Effects of fragrance on emotions: Moods and physiology.</a> Chem. Senses. 30, i248-i249 (2005).</li><li>Bredie, W. L. P., Tan, H. S. G., Wendin, K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+comparative+study+on+facially+expressed+emotions+in+response+to+basic+tastes.">A comparative study on facially expressed emotions in response to basic tastes.</a> Chem. Percept. 7 (1), 1-9 (2014).</li><li>Wendin, K., Allesen-Holm, B. H., Bredie, L. P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Do+facial+reactions+add+new+dimensions+to+measuring+sensory+responses+to+basic+tastes?.">Do facial reactions add new dimensions to measuring sensory responses to basic tastes?.</a> Food Qual Prefer. 22, 346-354 (2011).</li><li>Rosenstein, D., Oster, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Differential+facial+responses+to+four+basic+tastes+in+newborns.">Differential facial responses to four basic tastes in newborns.</a> Child Dev. 59 (6), 1555-1568 (1988).</li><li>Rosenstein, D., Oster, H., P, E. k. m. a. n., E, R. o. s. e. n. b. e. r. g. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Differential+facial+responses+to+four+basic+tastes+in+newborns.">Differential facial responses to four basic tastes in newborns.</a> What the face reveals: Basic and applied studies of spontaneous expression using the facial action coding system (FACS). , 302-327 (1997).</li><li>Rozin, P., Fallon, A. E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+perspective+on+disgust.">A perspective on disgust.</a> Psychol. Rev. 94 (1), 23-41 (1987).</li><li>Delarue, J., Blumenthal, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Temporal+aspects+of+consumer+preferences.">Temporal aspects of consumer preferences.</a> Curr. Opin. Food Sci. 3, 41-46 (2015).</li><li>Sudre, J., Pineau, N., Loret, C., Marin, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Comparison+of+methods+to+monitor+liking+of+food+during+consumption.">Comparison of methods to monitor liking of food during consumption.</a> Food Qual Prefer. 24 (1), 179-189 (2012).</li><li>Weiland, R., Ellgring, H., Macht, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Gustofacial+and+olfactofacial+responses+in+human+adults.">Gustofacial and olfactofacial responses in human adults.</a> Chem. Senses. 35 (9), 841-853 (2010).</li><li>Ekman, P., Cole, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Universal+and+cultural+differences+in+facial+expressions+of+emotion.">Universal and cultural differences in facial expressions of emotion.</a> Nebraska symposium on motivation. , 207-283 (1971).</li><li>Griemel, E., Macht, M., Krumhuber, E., Ellgring, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Facial+and+affective+reactions+to+tastes+and+their+modulation+by+sadness+and+joy.">Facial and affective reactions to tastes and their modulation by sadness and joy.</a> Physiol Behav. 89 (2), 261-269 (2006).</li></ol>

AFEA application in literature related to food and beverage is very limited1-11. The application to food is new, creating an opportunity for establishing methodology and data interpretation. Arnade (2013)7 found high individual variability among individual emotional response to chocolate milk and white milk using area under the curve analysis and analysis of variance. However, even with participant variability, participants generated a happy response longer while sad and disgusted had shorter time r...

Automated facial expression analysis (AFEA) is a prospective analytical tool for characterizing emotional responses to beverages and foods. Emotional analysis can add an extra dimension to existing sensory science methodologies, food evaluation practices, and hedonic scale ratings typically used both in research and industry settings. Emotional analysis could provide an additional metric that reveals a more accurate response to foods and beverages. Hedonic scoring may include participant bias due to failure to record reactions1.
AFEA research has been used in many research applications including computer gaming, user behavior, education/pedagogy, and psychology studies on empathy and deceit. Most food-associated research has focused on characterizing emotional response to food quality and human behavior with food. With the recent trend in gaining insights into food behaviors, a growing body of literature reports use of AFEA for characterizing the human emotional response associated with foods, beverages, and odorants1-12.
AFEA is derived from the Facial Action Coding System (FACS). The facial action coding system (FACS) discriminates facial movements characterized by action units (AUs) on a 5-point intensity scale13. The FACS approach requires trained review experts, manual coding, significant evaluation time, and provides limited data analysis options. AFEA was developed as a rapid evaluation method to determine emotions. AFEA software relies on facial muscular movement, facial databases, and algorithms to characterize the emotional response14-18. The AFEA software used in this study reached a &#34;FACS index of agreement of 0.67 on average on both the Warsaw Set of Emotional Facial Expression Pictures (WSEFEP) and Amsterdam Dynamic Facial Expression Set (ADFES), which is close to a standard agreement of 0.70 for manual coding&#34;19. Universal emotions included in the analysis are happy (positive), sad (negative), disgusted (negative), surprised (positive or negative), angry (negative), scared (negative) and neutral each on a separate scale of 0 to 1 (0=not expressed; 1=fully expressed)20. In addition, psychology literature includes happy, surprised, and angry as &#34;approach&#34; emotions (toward stimuli) and sad, scared, and disgusted as &#34;withdrawal&#34; emotions (away from aversive stimuli)21.
One limitation of the current AFEA software for characterizing emotions associated with foods is interference from facial movements associated with chewing and swallowing as well as other gross motor motions, such as extreme head movements. The software targets smaller facial muscular motions, relating position and degree of movement, based on over 500 muscle points on the face16,17. Chewing motions interfere with classification of expressions. This limitation may be addressed using liquefied foods. However, other methodology challenges can also decrease video sensitivity and AFEA analysis including data collection environment, technology, researcher instructions, participant behavior, and participant attributes.
A standard methodology has not been developed and verified for optimal video capture and data analysis using AFEA for emotional response to foods and beverages in a sensory evaluation laboratory setting. Many aspects can affect the video capture environment including lighting, shadowing due to lighting, participant directions, participant behavior, participant height, as well as, camera height, camera angling, and equipment settings. Moreover, data analysis methodologies are inconsistent and lack a standard methodology for assessing emotional response. Here, we will demonstrate our standard operating procedure for capturing emotional data and processing data into meaningful results using beverages (flavored milk, unflavored milk and unflavored water) for evaluation. To our knowledge only one peer reviewed publication, from our lab group, has utilized time series for data interpretation for emotions analysis8; however, the method has been updated for our presented method. Our aim is to develop an improved and consistent methodology to help with reproducibility in a sensory evaluation laboratory setting. For demonstration, the objective of the study model is to evaluate if AFEA could supplement traditional hedonic acceptability assessment of flavored milk, unflavored milk and unflavored water. The intention of this video protocol is to help establish AFEA methodology, standardize video capture criteria in a sensory evaluation laboratory (sensory booth setting), and illustrate a method for temporal emotional data analysis of a population.

<table border="0" cellpadding="0" cellspacing="0" width="1087">
	<tbody>
		<tr height="21">
			<td height="21" style="height:21px;width:216px;">2% Reduced Fat Milk</td>
			<td style="width:436px;">Kroger Brand, Cincinnati, OH or DZA Brands, LLC, Salisbury, NC</td>
			<td style="width:119px;">na</td>
			<td style="width:316px;">for solutions</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;">Drinking Water</td>
			<td>Kroger Brand, Cincinnati, OH</td>
			<td style="width:119px;">na</td>
			<td>for solutions</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;">Imitation Clear Vanilla Flavor</td>
			<td>Kroger Brand, Cincinnati, OH</td>
			<td style="width:119px;">na</td>
			<td>for solutions</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:216px;">Iodized Salt</td>
			<td>Kroger Brand, Cincinnati, OH</td>
			<td style="width:119px;">na</td>
			<td>for solutions</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:216px;">FaceReader 6</td>
			<td style="width:436px;">Noldus Information Technology, Wageningen, The Netherlands</td>
			<td style="width:119px;">na</td>
			<td>For Facial Analysis</td>
		</tr>
		<tr height="63">
			<td height="63" style="height:63px;width:216px;">Sensory Information Management System (SIMS) 2000</td>
			<td style="width:436px;">Sensory Computer Systems, Berkeley Heights, NJ</td>
			<td style="width:119px;">Version 6</td>
			<td>For Sensory Data Capture</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:216px;">Rhapsody</td>
			<td style="width:436px;">Acuity Brands Lighting, Inc., Conyers, GA</td>
			<td style="width:119px;"></td>
			<td>For Environment Illumination</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:216px;">R Version&#160;</td>
			<td style="width:436px;">R Core Team 2015</td>
			<td style="width:119px;">3.1.1</td>
			<td>For Statistical Analysis</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:216px;">Microsoft Office</td>
			<td style="width:436px;">Microsoft</td>
			<td style="width:119px;">na</td>
			<td>For Statistical Analysis</td>
		</tr>
		<tr height="42">
			<td height="42" style="height:42px;width:216px;">JMP</td>
			<td style="width:436px;">Statistical Analysis Software (SAS) Version 9.2, SAS Institute, Cary, NC</td>
			<td style="width:119px;">na</td>
			<td>For Statistical Analysis</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:216px;">Media Recorder 2.5</td>
			<td style="width:436px;">Noldus Information Technology, Wageningen, The Netherlands</td>
			<td style="width:119px;">na</td>
			<td>For capturing participants sensory evaluation</td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:216px;">Axis M1054 Camera</td>
			<td style="width:436px;">Axis Communications, Lund, Sweden</td>
			<td style="width:119px;">na</td>
			<td></td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:216px;">Beverage</td>
			<td style="width:436px;"></td>
			<td style="width:119px;">na</td>
			<td>Beverage or soft food for evaluation</td>
		</tr>
	</tbody>
</table>

protocol for data collection and analysis applied to automated facial expression analysis technology and temporal analysis for sensory evaluation

We demonstrate a method for capturing emotional response to beverages and liquefied foods in a sensory evaluation laboratory using automated facial expression analysis (AFEA) software. Additionally, we demonstrate a method for extracting relevant emotional data output and plotting the emotional response of a population over a specified time frame. By time pairing each participant's treatment response to a control stimulus (baseline), the overall emotional response over time and across multiple participants can be quantified. AFEA is a prospective analytical tool for assessing unbiased response to food and beverages. At present, most research has mainly focused on beverages. Methodologies and analyses have not yet been standardized for the application of AFEA to beverages and foods; however, a consistent standard methodology is needed. Optimizing video capture procedures and resulting video quality aids in a successful collection of emotional response to foods. Furthermore, the methodology of data analysis is novel for extracting the pertinent data relevant to the emotional response. The combinations of video capture optimization and data analysis will aid in standardizing the protocol for automated facial expression analysis and interpretation of emotional response data.

The method proposes a standard protocol for AFEA data collection. If suggested protocol steps are followed, unusable emotional data output (Figure 1) resulting from poor data collection (Figure 2: A; Left Picture) may be limited. Time series analysis cannot be utilized if log files (.txt) predominantly contain &#34;FIT_FAILED&#34; and &#34;FIND_FAILED&#34; as this is bad data (Figure 1). Furthermore, the method includes a protocol for dir...

Watch this Scientific Journal Video about Protocol for Data Collection and Analysis Applied to Automated Facial Expression Analysis Technology and Temporal Analysis for Sensory Evaluation at JoVE.com

Protocol for Data Collection and Analysis Applied to Automated Facial Expression Analysis Technology and Temporal Analysis for Sensory Evaluation

A protocol for capturing and statistically analyzing emotional response of a population to beverages and liquefied foods in a sensory evaluation laboratory using automated facial expression analysis software is described.

We demonstrate a method for capturing emotional response to beverages and liquefied foods in a sensory evaluation laboratory using automated facial ...

The overall goal of this methodology is to capture and analyze emotional responses to beverages and liquefied foods in a sensory evaluation laboratory using automated facial analysis software. This method can help answer key questions in the sensory science field such as how to measure a participant's emotional response to beverages and soft foods. The main advantage of this protocol is that it shows us standardized data capture methodology using automated facial expression analysis software.It also describes the method for analyzing automated facial expression output. The implications of this technique extend towards standardizing methodology of automated facial expression analysis as it often varies between laboratories. Although this method can provide insight to an individual's implicit emotional response, it can also be applied to other systems such as packaging, marketing, and labels.Lester will serve as a representative participant. In preparation for the experiment, create sample solutions by pouring half ounce aliquots of each original solution into two ounce transparent plastic sample cups, and then cap with color-coded lids. Next, escort the participant to the sensory booth and have them sit straight up against the back of the chair.Instruct the participant about the experimental process and standard operating procedures. Adjust the position of the chair and the camera angle so that the participants face is captured in the center of the video recording with no shadows on the chin or around the eyes. Instruct the participant to focus on the monitor display, follow the directions on the screen, and refrain from any sudden movements for 30 seconds after each sample is consumed.Demonstrate an impromptu practice session by first instructing the participant to hold up the associated identification card. Then instruct them to open the sample container, consume the entire beverage sample, and swallow. After consumption, instruct the participant to quickly move the sample cup away from the chin and place it on the table to avoid facial occlusion.Remind them to keep looking toward the monitor. Remind the participant that water will be provided on the serving tray to cleanse their palate in between samples when prompted by the monitor. Finally, instruct the participant to follow the instructions as they appear on the touch screen monitor.Confirm that the video camera is directed toward the participant's face by viewing the computer monitor on which the video capture is displayed. Then begin recording by clicking the record button on the computer monitor. Prompt the participant, via the touch screen monitor, to sip water to cleanse their palate.Place a unique-colored index card on top of each sample, allowing for identification of each treatment sample. Next, the touch screen monitor prompts the participant to hold up the associated index card pre-consumption for sample identification in the video, and then have them place the card back on the tray. Provide only one sample at a time on the serving tray, starting with a baseline sample of unflavored water.Have the participant consume the sample while facing the camera, and then wait approximately 30 seconds, a time enforced by the touch screen monitor. After 30 seconds, prompt the participant, via the touch screen monitor, to enter their hedonic acceptability score. Finally, have the participant rinse their mouth with drinking water before moving on to the next sample.Use facial analysis software to analyze the experimental videos. Add participants to the project by clicking the add participants icon. Press the batch analysis icon, which is the red and black target-like symbol, to analyze the project videos.Export the AFEA data output as log files for further analysis. Using the AFEA software, manually review each video and identify post-consumption time zero for each sample, which is defined as when the sample cup is below the participant's chin and longer occludes the face. Finally, record the timestamp in the data spreadsheet, and save this spreadsheet as a reference for extracting relevant data from videos.Unflavored milk, unflavored water, and vanilla extract flavor in milk were not different in mean acceptability scores. And salty flavor in milk had a lower mean hedonic acceptability score. However, AFEA time series analysis indicated unflavored milk generated less disgusted, surprised, sad, and neutral responses than did unflavored water.Furthermore, the vanilla extract flavor in milk introduced more happy and less sad expressions when compared to unflavored water. Salty flavor in milk generated more disgusted and less neutral responses when compared to unflavored water. Once mastered, this technique can be done in 30 minutes if it is performed properly.While attempting this procedure, it's important to remember to give the participant complete instructions and expectations. Following this procedure, other evaluations can be performed in order to answer additional questions like what is the influence on behavior and emotional response? After its development, this technique paved the way for researchers in the field of sensory science to explore implicit emotional responses in human participants to beverages and soft foods.After watching this video, you should have a good understanding of how to standardize the sensory lab environment, and how to instruct participants in order to capture the best data and results, as well as analyze automated facial expression analysis output. Don't forget that working with human participants can be sensitive. It is important to discuss ingredients prior to performing this procedure.

protocol-for-data-collection-analysis-applied-to-automated-facial

Research

JoVE Journal

Behavior

Contextual and Cued Fear Conditioning Test Using a Video Analyzing System in Mice

The contextual and cued fear conditioning test is one of the behavioral tests that assesses the ability of mice to learn and remember an association between environmental cues and aversive experiences. In this test, mice are placed into a conditioning chamber and are given parings of a conditioned stimulus (an auditory cue) and an aversive unconditioned stimulus (an electric footshock). After a delay time, the mice are exposed to the same conditioning chamber and a differently shaped chamber with presentation of the auditory cue. Freezing behavior during the test is measured as an index of fear memory. To analyze the behavior automatically, we have developed a video analyzing system using the ImageFZ application software program, which is available as a free download at http://www.mouse-phenotype.org/. Here, to show the details of our protocol, we demonstrate our procedure for the contextual and cued fear conditioning test in C57BL/6J mice using the ImageFZ system. In addition, we validated our protocol and the video analyzing system performance by comparing freezing time measured by the ImageFZ system or a photobeam-based computer measurement system with that scored by a human observer. As shown in our representative results, the data obtained by ImageFZ were similar to those analyzed by a human observer, indicating that the behavioral analysis using the ImageFZ system is highly reliable. The present movie article provides detailed information regarding the test procedures and will promote understanding of the experimental situation.

This article presents a protocol for a contextual and cued fear conditioning test using a video analyzing system to assess fear learning and memory in mice.

The contextual and cued fear conditioning test is one of the behavioral tests that assesses the ability of mice to learn and remember an association ...

Measurement of Fronto-limbic Activity Using an Emotional Oddball Task in Children with Familial High Risk for Schizophrenia

Adolescence is a critical developmental period where the early symptoms of schizophrenia frequently emerge. First-degree relatives of people with schizophrenia who are at familial high risk (FHR) may show similar cognitive and emotional changes. However, the neurological changes underlying the emergence of these symptoms remain unclear. This study sought to identify differences in frontal, striatal, and limbic regions in children and adolescents with FHR using functional magnetic resonance imaging. Groups of 21 children and adolescents at FHR and 21 healthy controls completed an emotional oddball task that relied on selective attention and the suppression of task-irrelevant emotional information. The standard oddball task was modified to include aversive and neutral distractors in order to examine potential group differences in both emotional and executive processing. This task was designed specifically to allow for children and adolescents to complete by keeping the difficulty and emotional image content age-appropriate. Furthermore, we demonstrate a technique for suitable fMRI registration for children and adolescent participants. This paradigm may also be applied in future studies to measure changes in neural activity in other populations with hypothesized developmental changes in executive and emotional processing.

This paper describes how to use the emotional oddball task and fMRI to measure brain activation in children and adolescents at familial high risk for schizophrenia (FHR).&#160;FMRI was used to measure differences in fronto-striato-limbic regions during an emotional oddball task. Children with FHR exhibited abnormal functional activation during adolescence.

Adolescence is a critical developmental period where the early symptoms of schizophrenia frequently emerge. First-degree relatives of people with ...

Recording Mouse Ultrasonic Vocalizations to Evaluate Social Communication

Mice emit ultrasonic vocalizations in different contexts throughout development and in adulthood. These vocal signals are now currently used as proxies for modeling the genetic bases of vocal communication deficits. Characterizing the vocal behavior of mouse models carrying mutations in genes associated with neuropsychiatric disorders such as autism spectrum disorders will help to understand the mechanisms leading to social communication deficits. We provide here protocols to reliably elicit ultrasonic vocalizations in pups and in adult mice. This standardization will help reduce inter-study variability due to the experimental settings. Pup isolation calls are recorded throughout development from individual pups isolated from dam and littermates. In adulthood, vocalizations are recorded during same-sex interactions (without a sexual component) by exposing socially motivated males or females to an unknown same-sex conspecific. We also provide a protocol to record vocalizations from adult males exposed to an estrus female. In this context, there is a sexual component in the interaction. These protocols are established to elicit a large amount of ultrasonic vocalizations in laboratory mice. However, we point out the important inter-individual variability in the vocal behavior of mice, which should be taken into account by recording a minimal number of individuals (at least 12 in each condition). These recordings of ultrasonic vocalizations are used to evaluate the call rate, the vocal repertoire and the acoustic structure of the calls. Data are combined with the analysis of synchronous video recordings to provide a more complete view on social communication in mice. These protocols are used to characterize the vocal communication deficits in mice lacking ProSAP1/Shank2, a gene associated with autism spectrum disorders. More ultrasonic vocalizations recordings can also be found on the mouseTube database, developed to favor the exchange of such data.

Mouse ultrasonic vocalizations are used as proxies to model the genetic bases of vocal communication deficits in mouse models for neuropsychiatric disorders. The present protocol describes three experimental contexts that reliably elicit ultrasonic vocalizations from pups (throughout development) and adult mice (same-sex interactions, male-estrus female interactions).

Mice emit ultrasonic vocalizations in different contexts throughout development and in adulthood. These vocal signals are now currently used as proxies ...

Measuring Attention and Visual Processing Speed by Model-based Analysis of Temporal-order Judgments

This protocol describes how to conduct temporal-order experiments to measure visual processing speed and the attentional resource distribution. The proposed method is based on a new and synergistic combination of three components: the temporal-order judgments (TOJ) paradigm, Bundesen&#39;s Theory of Visual Attention (TVA), and a hierarchical Bayesian estimation framework. The method provides readily interpretable parameters, which are supported by the theoretical and neurophysiological underpinnings of TVA. Using TOJs, TVA-based estimates can be obtained for a broad range of stimuli, whereas traditional paradigms used with TVA are mainly limited to letters and digits. Finally, the meaningful parameters of the proposed model allow for the establishment of a hierarchical Bayesian model. Such a statistical model allows assessing results in one coherent analysis both on the subject and the group level.
To demonstrate the feasibility and versatility of this new approach, three experiments are reported with attention manipulations in synthetic pop-out displays, natural images, and a cued letter-report paradigm.

Temporal-order judgments can be used to estimate processing speed parameters and attentional weights and thereby to infer the mechanisms of attentional processing. This methodology can be applied to a wide range of visual stimuli and works with many attention manipulations.

This protocol describes how to conduct temporal-order experiments to measure visual processing speed and the attentional resource distribution. The ...

Automated Analysis of a Nematode Population-based Chemosensory Preference Assay

The nematode, Caenorhabditis elegans' compact nervous system of only 302 neurons underlies a diverse repertoire of behaviors. To facilitate the dissection of the neural circuits underlying these behaviors, the development of robust and reproducible behavioral assays is necessary. Previous C. elegans behavioral studies have used variations of a "drop test", a "chemotaxis assay", and a "retention assay" to investigate the response of C. elegans to soluble compounds. The method described in this article seeks to combine the complementary strengths of the three aforementioned assays. Briefly, a small circle in the middle of each assay plate is divided into four quadrants with the control and experimental solutions alternately placed. After the addition of the worms, the assay plates are loaded into a behavior chamber where microscope cameras record the worms' encounters with the treated regions. Automated video analysis is then performed and a preference index (PI) value for each video is generated. The video acquisition and automated analysis features of this method minimizes the experimenter's involvement and any associated errors. Furthermore, minute amounts of the experimental compound are used per assay and the behavior chamber's multi-camera setup increases experimental throughput. This method is particularly useful for conducting behavioral screens of genetic mutants and novel chemical compounds. However, this method is not appropriate for studying stimulus gradient navigation due to the close proximity of the control and experimental solution regions. It should also not be used when only a small population of worms is available. While suitable for assaying responses only to soluble compounds in its current form, this method can be easily modified to accommodate multimodal sensory interaction and optogenetic studies. This method can also be adapted to assay the chemosensory responses of other nematode species.

We present a behavior recording setup and protocol that enables automated analysis of the nematode, Caenorhabditis elegans' preference for soluble compounds in a population-based assay. This article describes the construction of a behavior chamber, the behavioral assay protocol, and video analysis software usage.

The nematode, Caenorhabditis elegans' compact nervous system of only 302 neurons underlies a diverse repertoire of behaviors. To facilitate the dissection ...

Virtual Reality Experiments with Physiological Measures

Virtual reality (VR) experiments are increasingly employed because of their internal and external validity compared to real-world observation and laboratory experiments, respectively. VR is especially useful for geographic visualizations and investigations of spatial behavior. In spatial behavior research, VR provides a platform for studying the relationship between navigation and physiological measures (e.g., skin conductance, heart rate, blood pressure). Specifically, physiological measures allow researchers to address novel questions and constrain previous theories of spatial abilities, strategies, and performance. For example, individual differences in navigation performance may be explained by the extent to which changes in arousal mediate the effects of task difficulty. However, the complexities in the design and implementation of VR experiments can distract experimenters from their primary research goals and introduce irregularities in data collection and analysis. To address these challenges, the Experiments in Virtual Environments (EVE) framework includes standardized modules such as participant training with the control interface, data collection using questionnaires, the synchronization of physiological measurements, and data storage. EVE also provides the necessary infrastructure for data management, visualization, and evaluation. The present paper describes a protocol that employs the EVE framework to conduct navigation experiments in VR with physiological sensors. The protocol lists the steps necessary for recruiting participants, attaching the physiological sensors, administering the experiment using EVE, and assessing the collected data with EVE evaluation tools. Overall, this protocol will facilitate future research by streamlining the design and implementation of VR experiments with physiological sensors.

Virtual reality (VR) experiments can be difficult to implement and require meticulous planning. This protocol describes a method for the design and implementation of VR experiments that collect physiological data from human participants. The Experiments in Virtual Environments (EVE) framework is employed to accelerate this process.

Virtual reality (VR) experiments are increasingly employed because of their internal and external validity compared to real-world observation and ...

Biomechanical Analysis Methods to Assess Professional Badminton Players' Lunge Performance

Under the condition of simulating a badminton court in the laboratory, this study used the injury mechanism model to analyze the maximal right lunge movements of eight professional badminton players and eight amateur players. The purpose of this protocol is to study the differences in kinematics and joint moment of the right knee and ankle. A motion capture system and force plate were used to capture data of the joint movements of the lower extremity and the vertical ground reaction force (vGRF). Sixteen young men who did not have any sports injuries in the past 6 months took part in the study. The subjects performed a maximal right lunge from the start position with their right foot, stepping on and fully contacting with the force plate, hit the shuttlecock with an underhand stroke to the designated position in the backcourt, and then returned to the start/end position. All subjects wore the same badminton shoes to avoid a difference in impact from different badminton shoes. The amateur players showed a greater range of ankle movement and reverse joint moment on the frontal plane, and a larger internal joint rotation moment on the horizontal plane. The professional badminton players exhibited greater knee moment on the sagittal and frontal planes. Therefore, these factors should be considered in the development of the training program to reduce the risk of sports injuries in knee and ankle joints. This study simulates the real badminton court and calibrates the range of activities of each movement of the subjects so that the subjects complete the experimental action in a natural state with high quality. A limitation of this study is that it does not combine joint load and muscle activity. Another limitation is that the sample size is small and should be expanded in future studies. This research method can be applied to the lower limb biomechanical research of other footwork in the badminton project.

Here, we present a protocol to evaluate the differences in injury mechanisms between professional and amateur players when performing a badminton maximal right lunge movement by analyzing lower limb kinematics.

Under the condition of simulating a badminton court in the laboratory, this study used the injury mechanism model to analyze the maximal right lunge ...

Low-cost Protocol of Footprint Analysis and Hanging Box Test for Mice Applied the Chronic Restraint Stress

Gait disturbance is frequently observed in patients with movement disorders. In mouse models used for movement disorders, gait analysis is important behavioral test to determine whether the mice mimic the symptoms of patients. Motor deficits are often induced by stress when no spontaneous motor phenotype is observed in the mouse models. Therefore, gait analysis followed by stress loading would be a sensitive method for evaluating the motor phenotype in mouse models. However, researchers face the requirement of an expensive apparatus to obtain quantitative results automatically from gait analysis. For stress, stress loading by simple methods without expensive apparatuses required for electric shock and forced running is desirable. Therefore, we introduce a simple and low-cost protocol consisting of footprint analysis with paper and ink, hanging box test to evaluate motor function, and stress loading defined by restraint with a conical tube. The motor deficits of mice were successfully detected by this protocol.

The low-cost protocol consisting of footprint analysis and hanging box test after restraint stress is useful for evaluating the movement disorders of mouse model.

Gait disturbance is frequently observed in patients with movement disorders. In mouse models used for movement disorders, gait analysis is important ...

Modified Blood Collection from Tail Veins of Non-anesthetized Mice with a Vacuum Blood Collection System and Eyeglass Magnifier

Blood sample collection is the basis of experimental animal research. It is of importance to obtain adequate blood samples for various research purposes. The tail veins of mice are small, and it is sometimes difficult to obtain the required blood volume by conventional puncture methods. This study investigates the superiority of repeated blood sample collection from tail veins of mice through use of a vacuum blood collection system and eyeglass magnifier (experimental group) compared to conventional blood sampling methods (conventional group), performed by beginners and experts, respectively. With the help of an eyeglass magnifier, a butterfly needle tip is inserted into the tail vein of each mouse in the experimental group. When the vein is penetrated successfully, a blood sample is collected in the vacuum collection tube by insertion of the rubber end of a butterfly needle into the vacuum blood collection tube. The plunger is then used to collect blood without the help of the eyeglass magnifier in the conventional group. Success rates of blood sample collection by the beginners and experts were shown to be 70% vs. 100% (p &#60; 0.01) in the experimental group and 35% vs. 85% (p &#60; 0.01) in the conventional group. For both beginners and experts, puncture times required for obtaining required blood sample were significantly lower in the experimental group compared to the conventional group (2.40 &#177; 0.75 vs. 2.90 &#177; 0.31, p &#60; 0.05; 1.15 &#177; 0.37 vs. 1.55 &#177; 0.76, p &#60; 0.05). In conclusion, the presented blood sampling technique is feasible and easy to practice and enables frequent sampling of adequate blood volumes from non-anesthetized mice.&#160;

This study reports blood sampling from tail vein in mice using a vacuum extraction tube system with eyeglass magnifier. Our method is easy to practice and could be used for repeat blood sampling in mice.

Blood sample collection is the basis of experimental animal research. It is of importance to obtain adequate blood samples for various research purposes. ...

Using Facial Electromyography to Assess Facial Muscle Reactions to Experienced and Observed Affective Touch in Humans

"Affective" touch is believed to be processed in a manner distinct from discriminatory touch and to involve activation of C-tactile (CT) afferent fibers. Touch that optimally activates CT fibers is consistently rated as hedonically pleasant. Patient groups with impaired social-emotional functioning also show disordered affective touch ratings. However, relying on self-reported ratings of touch has many limitations, including recall bias and communication barriers. Here, we describe a methodological approach to study affective responses to touch via facial electromyography (EMG) that circumvents the reliance on self-report ratings. Facial EMG is an objective, quantitative, and non-invasive method to measure facial muscle activity indicative of affective responses. Responses can be assessed across healthy and patient populations without the need for verbal communication. Here, we provide two separate datasets demonstrating that CT-optimal and non-optimal touch elicit distinct facial muscle reactions. Moreover, facial EMG responses are consistent across stimulus modalities, e.g. tactile (experienced touch) and visual (observed touch). Finally, the temporal resolution of facial EMG can detect responses on timescales that supersede that of verbal reporting. Together, our data suggest that facial EMG is a suitable methodology for use in affective tactile research that can be used to supplement, or in some cases, supplant, existing measures.

We describe a protocol to assess facial muscle activity in response to experienced and observed tactile stimulation using facial electromyography.

"Affective" touch is believed to be processed in a manner distinct from discriminatory touch and to involve activation of C-tactile (CT) afferent fibers. ...