JoVE Logo
Faculty Resource Center

Sign In

Summary

Abstract

Introduction

Protocol

Representative Results

Discussion

Acknowledgements

Materials

References

Neuroscience

A Naturalistic Setup for Presenting Real People and Live Actions in Experimental Psychology and Cognitive Neuroscience Studies

Published: August 4th, 2023

DOI:

10.3791/65436

1Department of Cognitive Science, Middle East Technical University, 2Interdisciplinary Neuroscience Graduate Program, Bilkent University, 3Department of Computer Engineering, Middle East Technical University, 4Department of Cognitive Science, Jagiellonian University, 5Department of Psychology, Interdisciplinary Neuroscience Graduate Program, National Magnetic Resonance Research Center (UMRAM), Aysel Sabuncu Brain Research Center, Bilkent University

This study presents a naturalistic experimental setup that allows researchers to present real-time action stimuli, obtain response time and mouse tracking data while participants respond after each stimulus display, and change actors between experimental conditions with a unique system including a special transparent organic light-emitting diode (OLED) screen and light manipulation.

Perception of others' actions is crucial for survival, interaction, and communication. Despite decades of cognitive neuroscience research dedicated to understanding the perception of actions, we are still far away from developing a neurally inspired computer vision system that approaches human action perception. A major challenge is that actions in the real world consist of temporally unfolding events in space that happen "here and now" and are actable. In contrast, visual perception and cognitive neuroscience research to date have largely studied action perception through 2D displays (e.g., images or videos) that lack the presence of actors in space and time, hence these displays are limited in affording actability. Despite the growing body of knowledge in the field, these challenges must be overcome for a better understanding of the fundamental mechanisms of the perception of others' actions in the real world. The aim of this study is to introduce a novel setup to conduct naturalistic laboratory experiments with live actors in scenarios that approximate real-world settings. The core element of the setup used in this study is a transparent organic light-emitting diode (OLED) screen through which participants can watch the live actions of a physically present actor while the timing of their presentation is precisely controlled. In this work, this setup was tested in a behavioral experiment. We believe that the setup will help researchers reveal fundamental and previously inaccessible cognitive and neural mechanisms of action perception and will be a foundation for future studies investigating social perception and cognition in naturalistic settings.

A fundamental skill for survival and social interaction is the ability to perceive and make sense of others' actions and interact with them in the surrounding environment. Previous research in the last several decades has made significant contributions to understanding the fundamental principles of how individuals perceive and understand others' actions1,2,3,4,5,6,7,8,

Log in or to access full content. Learn more about your institution’s access to JoVE content here

The experimental protocols in this study were approved by the Ethics Committee for Research with Human Participants of Bilkent University. All participants included in the study were over 18 years old, and they read and signed the informed consent form before starting the study.

1. General design steps

NOTE: Figure 1A (top view) and Figure 1B and Figure 1C (front.......

Log in or to access full content. Learn more about your institution’s access to JoVE content here

Response time (RT) comparisons
The current study is an ongoing project, so, as representative results, data from the main part of the experiment (Experiment Part 3) are presented. These data are from 40 participants, including 23 females and 17 males, with ages ranging from 18-28 years (M = 22.75, SD = 3.12).

Investigating the extent of the normality of the distribution of the dependent variables was necessary in order to choose the appropriate statisti.......

Log in or to access full content. Learn more about your institution’s access to JoVE content here

The overarching goal of the present study is to contribute to our understanding of how human high-level visual perception and cognition work in real-life situations. This study focused on action perception and suggested a naturalistic yet controllable experimental paradigm that enables researchers to test how individuals perceive and evaluate others' actions by presenting real actors in a laboratory setting.

The significance of this proposed methodology compared to existing methodologies i.......

Log in or to access full content. Learn more about your institution’s access to JoVE content here

This work was supported by grants to Burcu A. Urgen from The Scientific and Technological Research Council of Türkiye (Project number: 120K913) and Bilkent University. We thank our pilot participant Sena Er Elmas for bringing the idea of adding background noise between the actor changes, Süleyman Akı for setting up the light circuit, and Tuvana Karaduman for the idea of using a security camera backstage and her contribution as one of the actors in the study.

....

Log in or to access full content. Learn more about your institution’s access to JoVE content here

Name Company Catalog Number Comments
Adjustable Height Table Custom-made N/A Width: 60 cm, Height: 62 cm, Depth: 40 cm
Ardunio UNO  Smart Projects A000066 Microcontroller used for switching the state of the LEDs from the script running on the operator PC
Black Pants No brand N/A Relaxed-fit pants of actors with no apparent brand name or logo.
Case Xigmatek EN43224 XIGMATEK HELIOS RAINBOW LED USB 3.0 MidT ATX GAMING CASE
CPU AMD YD1600BBAFBOX AMD Ryzen 5 1600 Soket AM4 3.2 GHz - 3.6 GHz 16 MB 65 W 12 nm Processor
Curtains Custom-made N/A Width: Part 1: 110 cm width from the wall (left) side, Part 2: 123 cm width above OLED display, Part 3: 170 cm from OLED display to right side, Cabin depth: 100 cm, Inside cabin depth: 100 cm, all heights 230 cm except for Part 2 (75 cm height)
Experimenter Adjustable/Swivel Chair No brand N/A Any brand
Experimenter Table Custom N/A Width: 160 cm, Height: 75 cm, Depth: 80 cm
GPU MSI GT 1030 2GHD4 LP OC MSI GEFORCE GT 1030 2GHD4 LP OC 2GB DDR4 64bit NVIDIA GPU
Grey-color blackout curtain Custom-made N/A Width: 330 cm, Height: 230 cm, used for covering the background
Hard Disk Kioxia LTC10Z240GG8 Kioxia 240 GB Exceria Sata 3.0 SSD (555 MB Read/540 MB Write)
Hard Disk Toshiba HDWK105UZSVA Toshiba 2,5'' 500 GB L200 SATA 3.0 8 MB Cache 5400 Rpm 7 mm Harddisk
High-Power MOSFET Module N/A N/A Heating Controller MKS MOSFET Module
Laptop Apple S/N: C02P916ZG3QT MacBook Pro 11.1 Intel Core i7 (Used as the actor PC)
Laptop Asus  UX410U Used for monitoring the security camera in real-time.
LED lights No brand N/A
LED Strip Power Supply No brand N/A AC to DC voltage converter used for supplying DC voltage to the lighting circuit
MATLAB  The MathWorks Inc., Natick, MA, USA Version: R2022a Used for programming the experiment.

Required Toolboxes:
MATLAB Support Package for Arduino Hardware (version 22.1.2)
Instrument Control Toolbox (version 4.6)
Psychtoolbox (version 3)
Monitor Philips UHB2051005145  Model ID: 242V8A/00, PHILIPS 23.8" 242V8A 4ms 75 Hz Freesync DP-HDMI+VGA IPS Gaming Monitor 
Motherboard MSI B450M-A PRO MAX MSI B450M-A PRO MAX Amd B450 Socket AM4 DDR4 3466(OC) M.2 Motherboard
Mouse Pad for participant Monster  78185721101502042 / 8699266781857 Pusat Gaming Mouse Pad XL
Night lamp Aukes ES620-0.5W 6500K-IP 20 Used for helping the actors see around when the lights are off in the backstage.
Participant Adjustable/Swivel Chair No brand N/A
Participant Table IKEA Sandsberg 294.203.93 Width: 110 cm, Height: 75 cm, Depth: 67 cm
Power Extension Cable Viko 9011760Y 250 V (6 inlets) Black
Power Extension Cable Viko 9011730Y 250 V (3 inlets) Black
Power Extension Cable Viko 9011330Y 250 V (3 inlets) White
Power Extension Cable s-link  Model No: SPG3-J-10 AC - 250 V 3 meter (5 inlets)
Power Supply THERMALTAKE PS-LTP-0550NHSANE-1 THERMALTAKE LITEPOWER RGB 550W APFC 12 cm FAN PSU
Professional Gaming Mouse Rampage 8680096 Model No: SMX-R50 
RAM GSKILL F4-3000C16S-8GVRB GSKILL 8GB (1x8GB) RipjawsV Red DDR4 3000 MHz CL16 1.35 V Single Ram
Reception bell No brand N/A Used for helping the communication between the experimenter and the actors.
Security Camera Brion Vega 2-20204210 Model:BV6000
Speakers Logitech P/N: 880-000-405 PID: WD528XM Used for playing the background music.
Survey Software Qualtrics  N/A
Switching Module No brand N/A F5305S PMOS Switch Module
Table under the OLED display Custom-made N/A Width: 123 cm, Height: 75 cm, Depth: 50 cm
Transparent OLED Display Planar PN: 998-1483-01 S/N:195210075 A 55-inch transparent display that showcases dynamic information, enabled the opaque and transparent usage during the experiment.
UPS EAG K200610100087 EAG 110
UPS EAG 210312030507 EAG 103
USB 2.0 Cable Type A/B for Arduino UNO (Blue) Smart Projects M000006  Used to connect the microcontroller to the experimenter PC.
USB to RS232 Converter  s-link 8680096082559 Model: SW-U610
White Long-Sleeved Blouse (2) H&M (cotton) N/A Relaxed-fit blouses with a round neckline and without ant apparent brand name or logo.
Wireless Keyboard Logitech P/N: 820-003488 S/N: 1719CE0856D8 Model: K360
Wireless Mouse Logitech S/N: 2147LZ96BGQ9 Model: M190 (Used as the response device)

  1. Grossman, E. D., Blake, R. Brain areas active during visual perception of biological motion. Neuron. 35 (6), 1167-1175 (2002).
  2. Saygin, A. P. Superior temporal and premotor brain areas necessary for biological motion perception. Brain. 130 (9), 2452-2461 (2007).
  3. Peelen, M. V., Downing, P. E. The neural basis of visual body perception. Nature Reviews Neuroscience. 8 (8), 636-648 (2007).
  4. Caspers, S., Zilles, K., Laird, A. R., Eickhoff, S. B. ALE meta-analysis of action observation and imitation in the human brain. Neuroimage. 50 (3), 1148-1167 (2010).
  5. Nelissen, K., et al. Action observation circuits in the macaque monkey cortex. Journal of Neuroscience. 31 (10), 3743-3756 (2011).
  6. Oosterhof, N. N., Tipper, S. P., Downing, P. E. Crossmodal and action-specific: Neuroimaging the human mirror neuron system. Trends in Cognitive Sciences. 17 (7), 311-318 (2013).
  7. Lingnau, A., Downing, P. E. The lateral occipitotemporal cortex in action. Trends in Cognitive Sciences. 19 (5), 268-277 (2015).
  8. Giese, M. A., Rizzolatti, G. Neural and computational mechanisms of action processing: Interaction between visual and motor representations. Neuron. 88 (1), 167-180 (2015).
  9. Tucciarelli, R., Wurm, M., Baccolo, E., Lingnau, A. The representational space of observed actions. eLife. 8, e47686 (2019).
  10. Tarhan, L., Konkle, T. Sociality and interaction envelope organize visual action representations. Nature Communications. 11 (1), 3002 (2020).
  11. Urgen, B. A., Saygin, A. P. Predictive processing account of action perception: Evidence from effective connectivity in the action observation network. Cortex. 128, 132-142 (2020).
  12. Newen, A., De Bruin, L., Gallagher, S. . The Oxford Handbook of 4E Cognition. , (2018).
  13. Snow, J. C., Culham, J. C. The treachery of images: How realism influences brain and behavior. Trends in Cognitive Sciences. 25 (6), 506-519 (2021).
  14. Matusz, P. J., Dikker, S., Huth, A. G., Perrodin, C. Are we ready for real-world neuroscience. Journal of Cognitive Neuroscience. 31 (3), 327-338 (2019).
  15. Zaki, J., Ochsner, K. The need for a cognitive neuroscience of naturalistic social cognition. Annals of the New York Academy of Sciences. 1167 (1), 16-30 (2009).
  16. Hasson, U., Honey, C. J. Future trends in Neuroimaging: Neural processes as expressed within real-life contexts. NeuroImage. 62 (2), 1272-1278 (2012).
  17. Risko, E. F., Laidlaw, K. E., Freeth, M., Foulsham, T., Kingstone, A. Social attention with real versus reel stimuli: toward an empirical approach to concerns about ecological validity. Frontiers in Human Neuroscience. 6, 143 (2012).
  18. Parsons, T. D. Virtual reality for enhanced ecological validity and experimental control in the clinical, affective and social neurosciences. Frontiers in Human Neuroscience. 9, 660 (2015).
  19. Deuse, L., et al. Neural correlates of naturalistic social cognition: brain-behavior relationships in healthy adults. Social Cognitive and Affective Neuroscience. 11 (11), 1741-1751 (2016).
  20. Camerer, C., Mobbs, D. Differences in behavior and brain activity during hypothetical and real choices. Trends in Cognitive Sciences. 21 (1), 46-56 (2017).
  21. Nastase, S. A., Goldstein, A., Hasson, U. Keep it real: Rethinking the primacy of experimental control in cognitive neuroscience. NeuroImage. 222, 117254 (2020).
  22. Kihlstrom, J. F. Ecological validity and "ecological validity&#34. Perspectives on Psychological Science. 16 (2), 466-471 (2021).
  23. Brunswik, E. . Perception and the Representative Design of Psychological Experiments. , (1956).
  24. Aronson, E., Carlsmith, J. M., Gilbert, D. T., Fiske, S. T., Lindzay, G. Experimentation in social psychology. The Handbook of Social Psychology. , 1-79 (1968).
  25. Ecological validity: Then and now. University of Colorado Available from: https://www.albany.edu/cpr/brunswik/notes/essay1.html (1998)
  26. Fan, S., Dal Monte, O., Chang, S. W. Levels of naturalism in social neuroscience research. IScience. 24 (7), 102702 (2021).
  27. Orban, G. A., Lanzilotto, M., Bonini, L. From observed action identity to social affordances. Trends in Cognitive Sciences. 25 (6), 493-505 (2021).
  28. Gray, H. M., Gray, K., Wegner, D. M. Dimensions of mind perception. Science. 315 (5812), 619 (2007).
  29. Li, Z., Terfurth, L., Woller, J. P., Wiese, E. Mind the machines: Applying implicit measures of mind perception to social robotics. 2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI. , 236-245 (2022).
  30. Karpinski, A., Steinman, R. B. The single category implicit association test as a measure of implicit social cognition. Journal of Personality and Social Psychology. 91 (1), 16 (2006).
  31. Greenwald, A. G., McGhee, D. E., Schwartz, J. L. Measuring individual differences in implicit cognition: the implicit association test. Journal of Personality and Social Psychology. 74 (6), 1464 (1998).
  32. Freeman, J. B., Ambady, N. MouseTracker: Software for studying real-time mental processing using a computer mouse-tracking method. Behavior Research Methods. 42 (1), 226-241 (2010).
  33. Pekçetin, T. N., Barinal, B., Tunç, J., Acarturk, C., Urgen, B. A. Studying mind perception in social robotics implicitly: The need for validation and norming. Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction. , 202-210 (2023).
  34. Yu, Z., Wang, F., Wang, D., Bastin, M. Beyond reaction times: Incorporating mouse-tracking measures into the implicit association test to examine its underlying process. Social Cognition. 30 (3), 289-306 (2012).
  35. Romero, C. A., Snow, J. C. Methods for presenting real-world objects under controlled laboratory conditions. Journal of Visualized Experiments. (148), e59762 (2019).
  36. Jastorff, J., Abdollahi, R. O., Fasano, F., Orban, G. A. Seeing biological actions in 3 D: An f MRI study. Human Brain Mapping. 37 (1), 203-219 (2016).
  37. Ferri, S., Pauwels, K., Rizzolatti, G., Orban, G. A. Stereoscopically observing manipulative actions. Cerebral Cortex. 26 (8), 3591-3610 (2016).
  38. Stangl, M., Maoz, S. L., Suthana, N. Mobile cognition: Imaging the human brain in the 'real world. Nature Reviews Neuroscience. 24 (6), 347-362 (2023).
  39. Kriegeskorte, N. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual Review of Vision Science. 1, 417-446 (2015).
  40. Marblestone, A. H., Wayne, G., Kording, K. P. Toward an integration of deep learning and neuroscience. Frontiers in Computational Neuroscience. 10, 94 (2016).

This article has been published

Video Coming Soon

JoVE Logo

Privacy

Terms of Use

Policies

Research

Education

ABOUT JoVE

Copyright © 2024 MyJoVE Corporation. All rights reserved