JoVE Logo
Faculty Resource Center

Sign In





Representative Results






Protocol for Data Collection and Analysis Applied to Automated Facial Expression Analysis Technology and Temporal Analysis for Sensory Evaluation

Published: August 26th, 2016



1Food Science and Technology, Virginia Tech, 2Civil and Environmental Engineering, Virginia Tech

A protocol for capturing and statistically analyzing emotional response of a population to beverages and liquefied foods in a sensory evaluation laboratory using automated facial expression analysis software is described.

We demonstrate a method for capturing emotional response to beverages and liquefied foods in a sensory evaluation laboratory using automated facial expression analysis (AFEA) software. Additionally, we demonstrate a method for extracting relevant emotional data output and plotting the emotional response of a population over a specified time frame. By time pairing each participant's treatment response to a control stimulus (baseline), the overall emotional response over time and across multiple participants can be quantified. AFEA is a prospective analytical tool for assessing unbiased response to food and beverages. At present, most research has mainly focused on beverages. Methodologies and analyses have not yet been standardized for the application of AFEA to beverages and foods; however, a consistent standard methodology is needed. Optimizing video capture procedures and resulting video quality aids in a successful collection of emotional response to foods. Furthermore, the methodology of data analysis is novel for extracting the pertinent data relevant to the emotional response. The combinations of video capture optimization and data analysis will aid in standardizing the protocol for automated facial expression analysis and interpretation of emotional response data.

Automated facial expression analysis (AFEA) is a prospective analytical tool for characterizing emotional responses to beverages and foods. Emotional analysis can add an extra dimension to existing sensory science methodologies, food evaluation practices, and hedonic scale ratings typically used both in research and industry settings. Emotional analysis could provide an additional metric that reveals a more accurate response to foods and beverages. Hedonic scoring may include participant bias due to failure to record reactions1.

AFEA research has been used in many research applications including computer gaming, user behavior, ed....

Log in or to access full content. Learn more about your institution’s access to JoVE content here

Ethics Statement: This study was pre-approved by Virginia Tech Institutional Review Board (IRB) (IRB 14-229) prior to starting the project.

Caution: Human subject research requires informed consent prior to participation. In addition to IRB approval, consent for use of still or video images is also required prior to releasing any images for print, video, or graphic imaging. Additionally, food allergens are disclosed prior to testing. Participants are asked prior to panel start if they have any intolerance, allergies or other concerns........

Log in or to access full content. Learn more about your institution’s access to JoVE content here

The method proposes a standard protocol for AFEA data collection. If suggested protocol steps are followed, unusable emotional data output (Figure 1) resulting from poor data collection (Figure 2: A; Left Picture) may be limited. Time series analysis cannot be utilized if log files (.txt) predominantly contain "FIT_FAILED" and "FIND_FAILED" as this is bad data (Figure 1). Furthermore, the method includes a protocol for dir.......

Log in or to access full content. Learn more about your institution’s access to JoVE content here

AFEA application in literature related to food and beverage is very limited1-11. The application to food is new, creating an opportunity for establishing methodology and data interpretation. Arnade (2013)7 found high individual variability among individual emotional response to chocolate milk and white milk using area under the curve analysis and analysis of variance. However, even with participant variability, participants generated a happy response longer while sad and disgusted had shorter time r.......

Log in or to access full content. Learn more about your institution’s access to JoVE content here

This project was funded, in part, by ConAgra Foods (Omaha, NE, USA), the Virginia Agricultural Experiment Station, the Hatch Program of the National Institute of Food and Agriculture, U.S. Department of Agriculture, and the Virginia Tech Water INTERface Interdisciplinary Graduate Education Program.


Log in or to access full content. Learn more about your institution’s access to JoVE content here

Name Company Catalog Number Comments
2% Reduced Fat Milk Kroger Brand, Cincinnati, OH or DZA Brands, LLC, Salisbury, NC na for solutions
Drinking Water Kroger Brand, Cincinnati, OH na for solutions
Imitation Clear Vanilla Flavor Kroger Brand, Cincinnati, OH na for solutions
Iodized Salt Kroger Brand, Cincinnati, OH na for solutions
FaceReader 6 Noldus Information Technology, Wageningen, The Netherlands na For Facial Analysis
Sensory Information Management System (SIMS) 2000 Sensory Computer Systems, Berkeley Heights, NJ Version 6 For Sensory Data Capture
Rhapsody Acuity Brands Lighting, Inc., Conyers, GA For Environment Illumination
R Version  R Core Team 2015 3.1.1 For Statistical Analysis
Microsoft Office Microsoft na For Statistical Analysis
JMP Statistical Analysis Software (SAS) Version 9.2, SAS Institute, Cary, NC na For Statistical Analysis
Media Recorder 2.5 Noldus Information Technology, Wageningen, The Netherlands na For capturing participants sensory evaluation
Axis M1054 Camera Axis Communications, Lund, Sweden na
Beverage na Beverage or soft food for evaluation

  1. De Wijk, R. A., Kooijman, V., Verhoeven, R. H. G., Holthuysen, N. T. E., De Graaf, C. Autonomic nervous system responses on and facial expressions to the sight, smell, and taste of liked and disliked foods. Food Qual Prefer. 26 (2), 196-203 (2012).
  2. De Wijk, R. A., He, W., Mensink, M. G. J., Verhoeven, R. H. G., De Graaf, C. ANS responses and facial expression differentiate between the taste of commercial breakfast drinks. PLoS ONE. 9 (4), 1-9 (2014).
  3. He, W., Boesveldt, S., De Graaf, C., De Wijk, R. A. Behavioural and physiological responses to two food odours. Appetite. 59 (2), 628 (2012).
  4. He, W., Boesveldt, S., De Graaf, C., De Wijk, R. A. Dynamics of autonomic nervous system responses and facial expressions to odors. Front Psychol. 5 (110), 1-8 (2014).
  5. Danner, L., Sidorkina, L., Joechl, M., Duerrschmid, K. Make a face! Implicit and explicit measurement of facial expressions elicited by orange juices using face reading technology. Food Qual Prefer. 32 (2014), 167-172 (2013).
  6. Danner, L., Haindl, S., Joechl, M., Duerrschmid, K. Facial expression and autonomous nervous system responses elicited by tasting different juices. Food Res Int. 64 (2014), 81-90 (2014).
  7. Arnade, E. A. . Measuring consumer emotional response to tastes and foods through facial expression analysis [thesis]. , 1-187 (2013).
  8. Leitch, K. A., Duncan, S. E., O'Keefe, S., Rudd, R., Gallagher, D. L. Characterizing consumer emotional response to sweeteners using an emotion terminology questionnaire and facial expression analysis. Food Res Int. 76, 283-292 (2015).
  9. Crist, C. A., et al. Application of emotional facial analysis technology to consumer acceptability using a basic tastes model. , (2014).
  10. Garcia-Burgos, D., Zamora, M. C. Facial affective reactions to bitter-tasting foods and body mass index in adults. Appetite. 71 (2013), 178-186 (2013).
  11. Garcia-Burgos, D., Zamora, M. C. Exploring the hedonic and incentive properties in preferences for bitter foods via self-reports, facial expressions and instrumental behaviours. Food Qual Prefer. 39 (2015), 73-81 (2015).
  12. Lewinski, P., Fransen, M. L., Tan, E. S. H. Predicting advertising effectiveness by facial expressions in response to amusing persuasive stimuli. J. Neurosci. Psychol. Econ. 7 (1), 1-14 (2014).
  13. Ekman, P., Friesen, W. V. Facial action coding system: A technique for the measurement of facial movement. , (1978).
  14. Viola, P., Jones, M. Rapid object detection using a boosted cascade of simple features. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern. 1, 511-518 (2001).
  15. Sung, K. K., Poggio, T. Example-based learning for view-based human face detection. IEEE Trans. Pattern Anal. Mach. Intell. 20 (1), 39-51 (1998).
  16. . . FaceReader 5™ Technical Specifications. , (2014).
  17. Cootes, T., Taylor, C. . Statistical models of appearance for computer vision: Technical report. , (2000).
  18. Bishop, C. M. . Neural networks for pattern recognition. , (1995).
  19. Lewinski, P., den Uyl, T. M., Butler, C. Automated facial coding: validation of basic emotions and FACS AUs in FaceReader. J. Neurosci. Psychol. Econ. 7 (4), 227-236 (2014).
  20. Noldus Information Technology. . FaceReader Reference Manual Version 6. , (2014).
  21. Alves, N. T., Fukusima, S. S., Aznar-Casanova, J. A. Models of brain asymmetry in emotional processing. Psychol Neurosci. 1 (1), 63-66 (2008).
  22. Costello, M., Clark, S., Clark, S., Costello, M., Drake, M., Bodyfelt, F. Preparation of samples for instructing students and staff in dairy products evaluation (Appendix F). The sensory evaluation of dairy foods. , 551-560 (2009).
  23. Porcherot, C., et al. How do you feel when you smell this? Optimization of a verbal measurement of odor-elicited emotions. Food Qual Prefer. 21, 938-947 (2010).
  24. Warrenburg, S. Effects of fragrance on emotions: Moods and physiology. Chem. Senses. 30, i248-i249 (2005).
  25. Bredie, W. L. P., Tan, H. S. G., Wendin, K. A comparative study on facially expressed emotions in response to basic tastes. Chem. Percept. 7 (1), 1-9 (2014).
  26. Wendin, K., Allesen-Holm, B. H., Bredie, L. P. Do facial reactions add new dimensions to measuring sensory responses to basic tastes?. Food Qual Prefer. 22, 346-354 (2011).
  27. Rosenstein, D., Oster, H. Differential facial responses to four basic tastes in newborns. Child Dev. 59 (6), 1555-1568 (1988).
  28. Rosenstein, D., Oster, H., P, E. k. m. a. n., E, R. o. s. e. n. b. e. r. g. Differential facial responses to four basic tastes in newborns. What the face reveals: Basic and applied studies of spontaneous expression using the facial action coding system (FACS). , 302-327 (1997).
  29. Rozin, P., Fallon, A. E. A perspective on disgust. Psychol. Rev. 94 (1), 23-41 (1987).
  30. Delarue, J., Blumenthal, D. Temporal aspects of consumer preferences. Curr. Opin. Food Sci. 3, 41-46 (2015).
  31. Sudre, J., Pineau, N., Loret, C., Marin, N. Comparison of methods to monitor liking of food during consumption. Food Qual Prefer. 24 (1), 179-189 (2012).
  32. Weiland, R., Ellgring, H., Macht, M. Gustofacial and olfactofacial responses in human adults. Chem. Senses. 35 (9), 841-853 (2010).
  33. Ekman, P., Cole, J. Universal and cultural differences in facial expressions of emotion. Nebraska symposium on motivation. , 207-283 (1971).
  34. Griemel, E., Macht, M., Krumhuber, E., Ellgring, H. Facial and affective reactions to tastes and their modulation by sadness and joy. Physiol Behav. 89 (2), 261-269 (2006).

This article has been published

Video Coming Soon

JoVE Logo


Terms of Use





Copyright © 2024 MyJoVE Corporation. All rights reserved