A subscription to JoVE is required to view this content. Sign in or start your free trial.

In This Article

  • Summary
  • Abstract
  • Introduction
  • Protocol
  • Results
  • Discussion
  • Disclosures
  • Acknowledgements
  • Materials
  • References
  • Reprints and Permissions

Summary

Here we present a protocol which characterizes the sense of agency developed over the control of sensate virtual or robotic prosthetic hands. Psychophysical questionnaires are employed to capture the explicit experience of agency, and time interval estimates (intentional binding) are employed to implicitly measure the sense of agency.

Abstract

This work describes a methodological framework that can be used to explicitly and implicitly characterize the sense of agency developed over the neural-machine interface (NMI) control of sensate virtual or robotic prosthetic hands. The formation of agency is fundamental in distinguishing the actions that we perform with our limbs as being our own. By striving to incorporate advanced upper-limb prostheses into these same perceptual mechanisms, we can begin to integrate an artificial limb more closely into the user's existing cognitive framework for limb control. This has important implications in promoting user acceptance, use, and effective control of advanced upper-limb prostheses. In this protocol, participants control a virtual prosthetic hand and receive kinesthetic sensory feedback through their preexisting NMIs. A series of virtual grasping tasks are performed and perturbations are systematically introduced to the kinesthetic feedback and virtual hand movements. Two separate measures of agency are employed: established psychophysical questionnaires (to capture the explicit experience of agency) and a time interval estimate task to capture the implicit sense of agency (intentional binding). Results of this protocol (questionnaire scores and time interval estimates) can be analyzed to quantify the extent of agency formation.

Introduction

As robotic prostheses become increasingly advanced, the importance of relevant sensory feedback will continue to grow. Sensory feedback affects how humans perceive, interact with, and even integrate machines into their body schema. Recent NMI techniques can now provide prosthetic limb users with intuitive control and achieve sensations associated with touch1,2,3,4,5,6,7 and kinesthesia (movement sense)8,9 in missing limbs. When this sensory information is paired with the visual information provided by watching the artificial limb during operation, we have access to key elements that inform the distinction of self-versus-other. Leveraging this access may help bring prosthetic limb users a step closer to operating an artificial limb as a part of their body, rather than just a tool.

Body awareness and the sense of being embodied arises from the establishment of agency (the experience of authorship over a limb's actions) and ownership (the feeling that a limb is a part of the body)10,11. Ownership is primarily mediated through the integration of touch and visual information12. Agency emerges from the integration of intent, movement sensation (kinesthesia), visual information, and predictive cognitive models11. During the performance of a voluntary action, agency is formed when the sensory consequences of that action align with the performer's intent and predictions from the performer's internal models13. Agency is separate and distinct from ownership. The concept of limb ownership has been studied frequently in prosthesis literature14. A sense of limb ownership forms in NMI participants when touch feedback is spatially and temporally appropriate, as measured explicitly through questionnaires or implicitly through changes in residual limb temperature, or temporal order judgments15. However, fewer opportunities have existed to explore agency in the context of NMI16. Recent work with NMI participants has demonstrated that agency can be purposefully promoted and is separate from the experience of ownership8.

Agency is particularly significant in the operation of robotic prostheses as it is a cognitive link to the control of the artificial limb's physical actions through experiences of causality, the feeling of controlling the artificial limb or causing something to happen17. Robotic prostheses are advanced computerized machines that the user must cooperate with to effectively complete tasks. Some prosthetic limbs have incorporated autonomous functions, such as grip-slip detection and correction; yet these systems have seen limited adoption as functionality running outside the user's control can be viewed as frustrating if not appropriately implemented8,18. This presents a fundamental challenge that is echoed throughout applications of human cooperation with autonomous machines. That is, humans often trust their own actions over those resulting from a collaboration with computers or machines, and this trust directly influences an operator's likelihood to use the autonomous functions19,20. As humans, we innately trust ourselves and our bodies to perform the actions we intend; when this is achieved, we establish an intrinsic sense of agency. Interestingly, the formation of agency is impacted in cooperative human-computer actions. During human-human cooperative tasks, a shared sense of agency may be formed over movement21; yet, the literature suggests that shared agency is encumbered during human-computer cooperation22,23. These challenges are reflected in prosthetic upper-limb use, and the rejection rates of robotic devices remain high, with 23% - 39% of users discontinuing their use24. In fact, many prosthesis users still prefer body-powered systems25. These systems remove the computerized machine from the control loop and more intimately couple the user's body movement to the prosthesis movement via wire cables. This further reinforces the importance of cognitive integration in the use of advanced prosthetic devices. We suggest that NMI systems can provide a number of the necessary sensory and motor pieces to help move artificial limbs closer to establishing a cooperative sense of agency, and this will be instrumental in promoting the acceptance and the true integration of these computerized machines with their users.

Agency may be measured in a number of ways. The simplest measures use psychophysical questionnaires or scales that explicitly ask participants to whom or what they attribute an event17,26,27. This relies on an individual's existing perception of "self" by requiring participants to make inferential judgments of self-attribution (i.e., to explicitly judge whether "I" or another entity was responsible for an action or event). Implicit measures provide insight into the background cognitive processes that occur during motor action and sensory events. This view of agency attempts to measure that which is not explicitly perceived by an individual. Typically this is achieved by having participants characterize a perceived difference in self- and externally-generated actions, for example having participants report the duration of time they perceived to occur between a self- and externally-generated event17,28. During the performance of self-generated actions, agency implicitly manifests as a perceptual compression in time between actions and their sensory consequences, known as intentional binding28. When individuals report the time they perceived to occur between an action and its outcome, a shorter perceived duration of time corresponds to a more strongly formed sense of agency29,30. Interestingly, it has been demonstrated that explicit and implicit measures may not directly correlate as they are likely characterizing different perceptual mechanisms17 that together inform the sense of agency. As such, establishing a more comprehensive understanding of agency formation during prosthesis use will likely require experimental protocols employing both explicit and implicit measures.

This work describes a methodological framework that can be used to explicitly and implicitly characterize the sense of agency developed over the NMI control of sensate virtual or robotic prosthetic hands. Two techniques to measure agency during the performance of a sensorimotor object-grasping task are highlighted. Established psychophysical questionnaires are employed to capture the explicit experience of agency, while time interval estimates (intentional binding) are employed to implicitly measure the sense of agency.

The scope of this protocol is to evaluate the sense of agency in the context of an NMI that provides physiologically relevant active motor control and kinesthetic feedback. These techniques are generalizable to virtual or physical prosthetic NMI systems. There are minimal restrictions on the populations that may be recruited to perform this protocol. For instance, the mobility of the participant's upper limbs cannot be bilaterally affected (they must have one sound limb), and they must possess the cognitive ability to make time-based judgments and articulate experienced sensations.

Protocol

This protocol has been previously approved and follows the guidelines of the Cleveland Clinic’s human research ethics committee.

1. Hardware and Software of the NMI

  1. Establish each individual participant’s NMI control and feedback so that when they attempt to perform a movement, they see and feel a virtual prosthesis complete that movement.
    1. Generate a hand kinesthetic percept through the participant’s NMI and capture the kinematics of the perceived motion by having the participant demonstrate what they feel using their intact hand.
      NOTE: Techniques to characterize kinesthetic percept kinematics have been illustrated in other works8 and may be achieved using a data glove or an optical motion capture system.
    2. Use a virtual hand/prosthetic simulation to reproduce the kinematics of the movement percept.
    3. Set up hardware to capture the intentional hand movement control signals from the participant’s NMI.
    4. Map this control signal to the activity of the virtual prosthesis.
    5. Create a master control program that coordinates the acquisition of the NMI control signal, the movement of the virtual prosthesis, and the generation of kinesthetic NMI feedback in real-time.

2. Experimental Setup

  1. Seat the participant and position a monitor horizontally (i.e., on its back, facing upward) on a table in front of them.
  2. Display the virtual prosthesis on the monitor and adjust its size and location so that it is positioned congruently with the location of their missing limb.
  3. Render objects (e.g., floating balls) in the virtual environment to serve as stop points for the close and open positions of the hand (endpoints of movement).
  4. Configure the master control program so that when the virtual digits make contact with the virtual stop points, an auditory tone is played after an adjustable time delay (300, 500, 700, or 1,000 ms).

3. Experimental Conditions

  1. Build an input file for the master control program that specifies the settings for each trial, including the auditory tone delay, whether the NMI feedback is turned on/off, the speed and direction of the virtual hand movement, and the delay between the command and the virtual hand movement.
    1. Create two control conditions, a baseline and a passive condition.
      1. For the baseline condition, configure the kinematics and control of the virtual hand to match the NMI kinesthetic percept.
        NOTE: The baseline condition represents the ideal congruency of motor intent, movement kinematics, and kinesthetic feedback.
      2. Program the passive condition to perform a virtual hand movement when triggered by the investigator (removing the control from the user) while still providing the participant with the NMI kinesthetic percept.
        NOTE: The passive condition captures the theoretical worst-case agency conditions (i.e., movement in the absence of control [without intent], similar to one’s body being passively moved).
    2. Program additional conditions designed to parse out the contributions to agency of motor intent, kinesthetic sensation, and temporal mismatch with the displayed kinematics of the virtual prosthesis. Consider using the following five conditions.
      1. Opposite movement: the NMI kinesthetic feedback indicates that the hand closes while the hand visualization opens.
      2. Too fast: the hand visualization closes faster than indicated by the NMI kinesthetic feedback.
      3. Too slow: the hand visualization closes slower than indicated by the NMI kinesthetic feedback.
      4. Onset delay: the hand visualization closes 1 s later than indicated by the NMI kinesthetic feedback.
      5. No feedback: the hand visualization closes without any NMI kinesthetic feedback.

4. Performance of the Experiment

  1. Instruct participants to drive the hand from the open to the closed position without stopping and to report their estimation of the time delay from when the virtual digits contacted the virtual stop points to when they heard the auditory tone.
    NOTE: Participants may use any representation of time between 0 and 1 s that makes the most sense to them (e.g., milliseconds, fractions of seconds, a 0 - 10 scale).
  2. Initiate each trial by pressing a start button on the master control program, which moves the virtual hand to the start position, signaling the beginning of the trial. This cues the participant to drive the virtual hand to the virtual stop points, which causes an auditory tone to play after a randomized delay (300, 500, or 700 ms).
    1. Record the participant’s verbally reported estimation of the time delay interval.
  3. Organize trials into experimental blocks.
    1. Begin with two practice sessions and exclude them from the final analysis.
      1. In the first practice session, have the participant drive the hand to the movement endpoint and play the auditory tone 1,000 ms after the virtual digits reach the virtual stop points for 10 trials.
        1. Participants do not need to report the estimated intervals for this practice session.
          NOTE: This step is necessary to orient the participants to how long a single second feels.
      2. In the second practice session, again, have the participant drive the hand to the movement endpoint. Randomize the auditory tones so that the 300, 500, and 700 ms delay intervals are presented at least 5x each.
        1. Ask participants to report the estimated delay intervals.
        2. Do not inform the participant of how close their estimates of the delay intervals are to the actual delay during these practice trials or subsequent trials in the experimental block.
          NOTE: This step is important as participants will likely be inexperienced in making time judgments on a scale of fractions of a second, and the testing procedure may not be intuitive to the unpracticed test participant.
    2. Move to experimental sets of 15 trials for each condition. Present the conditions in a randomized order and administer a questionnaire at the end of each condition.
      1. Instruct the participants to reflect upon the latest set of trials and complete the eight-statement agency questionnaire (includes four questions to quantify the explicit experience of agency and four control questions [example provided in the Supplementary File])8,26.
        1. Randomize the questionnaire statements to provide at least five unique question orders to be randomly presented to the participants.
    3. End the experimental block with a set of 15 trials for the passive condition and administer a questionnaire after completing these trials.
      NOTE: Administer the passive trials at the end of each experimental block to avoid interfering with an established sense of agency.
  4. Complete four experimental blocks with different randomized orders of experimental conditions.
  5. Provide multiple opportunities over the duration of testing for the participants to take a break. There is no minimum time or time limit for these breaks, but ensure the participant is not physically or mentally fatigued prior to continuing the testing.

Results

The experimental protocol was performed with three amputee participants operating a sensate virtual prosthesis via their NMI8 (Figure 1). The setup used a participant-controllable virtual hand moving through preprogrammed kinematic profiles using the MuJoCo HAPTIX physics engine31. The virtual hand was displayed on a horizontal monitor in front of the participants at a location spatially congruent with ...

Discussion

Here a methodological framework is presented to characterize the experience of agency formed while operating sensate prostheses via NMIs. In this context, agency is particularly relevant as it bridges physical action to the background cognitive processes that shape perception. Through a participant's prosthesis and NMI, we have direct access to a number of key elements that establish the sense of agency: intent, motor output, and movement sensation. Of importance to advanced prosthetic limb control, the tool...

Disclosures

The authors have nothing to disclose.

Acknowledgements

The authors would like to thank Madeline Newcomb for her contributions to the figure generation. This work was funded by the U.S. taxpayers through an NIH, Office of the Director, Common Fund, Transformative R01 Research Award (grant #1R01NS081710-01) and the Defense Advanced Research Projects Agency (contract number N66001-15-C-4015 under the auspices of Biology Technology Office program manager D. Weber).

Materials

NameCompanyCatalog NumberComments
LabVIEW 2015, Service Pack 1, Version 15.0.1f2 64-bitNational Instruments, Austin, TX, USAFull or Pro VersionWe wrote custom software in LabVIEW to coordinate virtual prosthesis control with kinesthetic feedback as well as to present experimental conditions and record data.
8-Slot, USB CompactDAQ ChassisNational Instruments, Austin, TX, USAcDAQ-9178
±60 V, 800 kS/s, 12-Bit, 8-Channel C Series Voltage Input ModuleNational Instruments, Austin, TX, USANI-9221
100 kS/s/ch Simultaneous, ±10 V, 4-Channel C Series Voltage Output ModuleNational Instruments, Austin, TX, USANI-9263
Custom Wearable Kinesthetic TactorHDT Global, Solon, OH, USAN/AThis item was custom made. Other methods of delivering kinesthetic feedback are acceptable as long as the participant feels the sensation of the hand moving in real-time with the movements of the virtual hand.
MuJoCo Physics Engine, HAPTIX VersionRoboti LLC, Redmond, WA, USAmjhaptix150Newer versions of MuJoCo should be acceptable as well. We used the MPL Gripper Model.
Myobock Electrodes, powered by Otto Bock EnergyPack in MyoBoy Battery ReceptacleOttobock, Duderstadt, Germanyelectrodes: 13E200=60
battery: 757B21
battery receptacle: 757Z191=2
Any setup that provides an amplified, filtered, and rectified EMG or neural control signal could be used.

References

  1. Kuiken, T. A., Marasco, P. D., Lock, B. A., Harden, R. N., Dewald, J. P. A. Redirection of cutaneous sensation from the hand to the chest skin of human amputees with targeted reinnervation. Proceedings of the National Academy of Sciences. 104 (50), 20061-20066 (2007).
  2. Hebert, J. S., et al. Novel targeted sensory reinnervation technique to restore functional hand sensation after transhumeral amputation. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 22 (4), 765-773 (2014).
  3. Tan, D. W., et al. A neural interface provides long-term stable natural touch perception. Science Translational Medicine. 257 (6), (2014).
  4. Oddo, C. M., et al. Intraneural stimulation elicits discrimination of textural features by artificial fingertip in intact and amputee humans. eLife. 5 (MARCH2016), (2016).
  5. Raspopovic, S., et al. Bioengineering: Restoring natural sensory feedback in real-time bidirectional hand prostheses. Science Translational Medicine. 6 (222), (2014).
  6. Flesher, S. N., et al. Intracortical microstimulation of human somatosensory cortex. Science Translational Medicine. 8 (361), (2016).
  7. Tabot, G. A., et al. Restoring the sense of touch with a prosthetic hand through a brain interface. Proceedings of the National Academy of Sciences. 110 (45), 18279-18284 (2013).
  8. Marasco, P. D., et al. Illusory movement perception improves motor control for prosthetic hands. Science Translational Medicine. 10 (432), (2018).
  9. Horch, K., Meek, S., Taylor, T. G., Hutchinson, D. T. Object discrimination with an artificial hand using electrical stimulation of peripheral tactile and proprioceptive pathways with intrafascicular electrodes. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 19 (5), 483-489 (2011).
  10. Braun, N., et al. The senses of agency and ownership: A review. Frontiers in Psychology. 9 (APR), (2018).
  11. Van Den Bos, E., Jeannerod, M. Sense of body and sense of action both contribute to self-recognition. Cognition. 85 (2), 177-187 (2002).
  12. Botvinick, M., Cohen, J. Rubber hands "feel" touch that eyes see. Nature. 391 (6669), 756 (1998).
  13. Gallagher, S. Philosophical conceptions of the self: Implications for cognitive science. Trends in Cognitive Sciences. 4 (1), 14-21 (2000).
  14. Niedernhuber, M., Barone, D. G., Lenggenhager, B. Prostheses as extensions of the body: Progress and challenges. Neuroscience & Biobehavioral Reviews. 92, 1-6 (2018).
  15. Marasco, P. D., Kim, K., Colgate, J. E., Peshkin, M. A., Kuiken, T. A. Robotic touch shifts perception of embodiment to a prosthesis in targeted reinnervation amputees. Brain. 134 (3), 747-758 (2011).
  16. Rognini, G., Blanke, O. Cognetics: Robotic Interfaces for the Conscious Mind. Trends in Cognitive Sciences. 20 (3), 162-164 (2016).
  17. Dewey, J. A., Knoblich, G. Do implicit and explicit measures of the sense of agency measure the same thing. PLoS ONE. 9 (10), (2014).
  18. Edwards, A. L. . Adaptive and Autonomous Switching: Shared Control of Powered Prosthetic Arms Using Reinforcement Learning. , (2016).
  19. Desai, M., Stubbs, K., Steinfeld, A., Yanco, H. Creating trustworthy robots: Lessons and inspirations from automated systems. Adaptive and Emergent Behaviour and Complex Systems - Proceedings of the 23rd Convention of the Society for the Study of Artificial Intelligence and Simulation of Behaviour, AISB 2009. , 49-56 (2009).
  20. Lee, J. D., See, K. A. Trust in automation: designing for appropriate reliance. Human Factors. 46 (1), 50-80 (2004).
  21. Moore, J. W. What is the sense of agency and why does it matter?. Frontiers in Psychology. 7 (AUG), 1-9 (2016).
  22. Obhi, S. S., Hall, P. Sense of agency in joint action: Influence of human and computer co-actors. Experimental Brain Research. 211 (3-4), 663-670 (2011).
  23. Sahaï, A., Pacherie, E., Grynszpan, O., Berberian, B. Co-representation of human-generated actions vs. machine-generated actions: Impact on our sense of we-Agency?. 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). , (2017).
  24. Biddiss, E., Chau, T. Upper limb prosthesis use and abandonment: A survey of the last 25 years. Prosthetics and Orthotics International. 31 (3), 236-257 (2007).
  25. Atkins, D. J., Heard, D. C. Y., Donovan, W. H. Epidemiologic overview of individuals with upper-limb loss and their reported research priorities. Journal of Prosthetics and Orthotics. 8 (1), 2-11 (1996).
  26. Kalckert, A., Ehrsson, H. H. Moving a Rubber Hand that Feels Like Your Own: A Dissociation of Ownership and Agency. Frontiers in Human Neuroscience. 6 (March), 1-14 (2012).
  27. Caspar, E. A., Cleeremans, A., Haggard, P. The relationship between human agency and embodiment. Consciousness and Cognition. 33, 226-236 (2015).
  28. Haggard, P., Clark, S., Kalogeras, J. Voluntary action and conscious awareness. Nature Neuroscience. 5 (4), 382-385 (2002).
  29. Engbert, K., Wohlschläger, A., Haggard, P. Who is causing what? The sense of agency is relational and efferent-triggered. Cognition. 107 (2), 693-704 (2008).
  30. Moore, J. W., Wegner, D. M., Haggard, P. Modulating the sense of agency with external cues. Consciousness and Cognition. 18 (4), 1056-1064 (2009).
  31. Kumar, V., Todorov, E. MuJoCo HAPTIX: A virtual reality system for hand manipulation. 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids). , (2015).
  32. Kuiken, T. A., et al. Targeted reinnervation for enhanced prosthetic arm function in a woman with a proximal amputation: a case study. Lancet. 369 (9559), 371-380 (2007).

Reprints and Permissions

Request permission to reuse the text or figures of this JoVE article

Request Permission

Explore More Articles

Sense Of AgencyNeural machine InterfaceProsthesesCognitive Perceptual CommunicationFeedback EffectsMyoelectric ProsthesesHand Kinesthetic PerceptNMI Control SignalVirtual Prosthesis SimulationMaster Control ProgramKinematics Of MovementAuditory FeedbackMotor IntentKinesthetic Sensation

This article has been published

Video Coming Soon

JoVE Logo

Privacy

Terms of Use

Policies

Research

Education

ABOUT JoVE

Copyright © 2025 MyJoVE Corporation. All rights reserved