This research was funded by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation: project No. 222641018-SFB/TRR 135 TP C1 and IRTG-1901 &#34;The Brain in Action&#34;) and by the Research Cluster &#34;The Adaptive Mind&#34; funded by the Excellence Program of the Hessian Ministry of Higher Education, Science, Research, and Art. The authors thank the Qualisys support team, including Mathias Bankay and Jeffrey Thingvold, for assistance in developing our methods. The authors also thank Michaela Jeschke for posing as the hand model. All data and analysis scripts to reproduce the method and the results presented in the manuscript are available on Zenodo (doi: 10.5281/zenodo.7458911).

Prior to beginning an experiment, the participants must provide informed consent in accordance with the institutional guidelines and the Declaration of Helsinki. All the protocols described here have been approved by the local ethics committee of Justus Liebig University Giessen (LEK-FB06).
1. Installation of all the necessary software
<ol>
	<li>Download the project repository at Data and Code Repository.</li>
	<li>Install the software listed in the Table of Materials (note the software versions and follow the links for purchase options and instructions).</li>
	<li>Within the Data and Code Repository, open a command window, and run the following command: 
	conda env create -f environment.yml</li>
	<li>Download and install the pretrained DeepHandMesh29 instantiation following the instructions provided at&#160;https://github.com/facebookresearch/DeepHandMesh.
	<ol>
		<li>Place DeepHandMesh in the folder &#34;deephandmesh&#34; of the Data and Code Repository. Replace the file &#34;main/model.py&#34; with the model.py file contained in the Data and Code Repository.</li>
	</ol>
	</li>
</ol>
2. Preparing the motion capture system
<ol>
	<li>Position a workbench within a tracking volume imaged from multiple angles by motion-tracking cameras arranged on a frame surrounding the workspace (Figure 1A). Prepare reflective markers by attaching double-sided adhesive tape to the base of each marker.</li>
	<li>Execute Qualisys Track Manager (QTM) as an Administrator. 
	NOTE: Executing QTM as an Administrator is necessary for the Python SDK to take control of the QTM interface. We advise always running QTM as an Administrator.</li>
</ol>
3. Calibrating the cameras
<ol>
	<li>Place the L-shaped calibration object within the tracking volume.</li>
	<li>Within the QTM, click on Calibrate in the Capture menu, or press the wand icon in the Capture toolbar. Wait for a calibration window to open. Select the duration of the calibration, and press OK.</li>
	<li>Wave the calibration wand across the tracking volume for the duration of the calibration. Press the Export button, and specify a file path in which to export the calibration as a text file. Accept the calibration by pressing OK.</li>
</ol>
4. Creating a stimulus object
<ol>
	<li>Construct a virtual 3D object model in the form of a polygon mesh. Use a 3D printer to construct a physical replica of the object model. 
	&#8203;NOTE: The data repository in step 1.1 provides example objects in STL and Wavefront OBJ file formats. Objects in STL format are manifold and ready for 3D printing.</li>
</ol>
5. Preparing the stimulus object
<ol>
	<li>Attach four non-planar reflective markers to the surface of the real object. Place the object within the tracking volume.</li>
	<li>In the project repository, execute the Python script &#34;Acquire_Object.py&#34;. Follow the instructions provided by the script to perform a 1 s capture of the 3D position of the object markers.</li>
	<li>Select all the markers of the rigid body. Right-click on and select Define Rigid Body (6DOF) | Current Frame. Enter the name of the rigid body, and press OK.</li>
	<li>In the File menu, select Export | To TSV. In the new window, check the boxes 3D, 6D, and Skeleton in the Data Type settings. Check all the boxes in the General settings. Press OK and then Save.</li>
</ol>
6. Co-registering real and mesh model versions of the stimulus object
<ol>
	<li>Open Blender, and navigate to the Scripting workspace. Open the file &#34;Object_CoRegistration.py&#34;, and press Run. Navigate to the Layout workspace, and press n to toggle the sidebar. Within the sidebar, navigate to the Custom tab.</li>
	<li>Select the .obj file to be co-registered, and press the Load Object button.</li>
	<li>Select the trajectory file that was exported in step 3.3, and specify the names of the markers attached to the rigid object separated by semicolons. In the Marker header, specify the line in the trajectory file that contains the column names of the data (counting starts at 0).</li>
	<li>Select the corresponding rigid body file with the 6D suffix, and specify the name of the rigid body defined in step 4.1. In the 6D header, specify the line in the rigid body file that contains the column names of the data.</li>
	<li>Press Load Markers. Translate and rotate the Markers object and/or the Object object to align them. Specify a mesh output file, and press Run Coregistration. This will output an .obj file that contains the co-registered stimulus mesh.</li>
</ol>
7. Setting up markers on the hands
<ol>
	<li>Attach 24 spherical reflective markers on different landmarks of a participant&#39;s hand using double-sided tape. 
	NOTE: The specific positioning of the markers is demonstrated in Figure 2.
	<ol>
		<li>Position the markers centrally on top of the respective fingertips, as well as the distal interphalangeal joints, proximal interphalangeal joints, and metacarpophalangeal joints of the index finger, middle finger, ring finger, and small finger.</li>
		<li>For the thumb, position one marker each on the fingertip and the basal carpometacarpal joint, as well as a pair of markers each on the metacarpophalangeal and the interphalangeal joints. 
		&#8203;NOTE: These marker pairs need to be displaced in opposite directions perpendicular to the thumb&#39;s main axis and are necessary to estimate the thumb&#39;s orientation.</li>
		<li>Finally, place markers at the center of the wrist and on the scaphotrapeziotrapezoidal joint.</li>
	</ol>
	</li>
</ol>
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/64877/64877fig2v2.jpg" /> 
Figure 2: Marker placement on a participant&#39;s hand. Abbreviation: RH = right hand. <a href="https://www.jove.com/files/ftp_upload/64877/64877fig02large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
8. Acquiring a single trial
<ol>
	<li>Ask the participant to place their hand flat on the workbench with the palm facing downward and to close their eyes. Place the stimulus object on the workbench in front of the participant.</li>
	<li>While the QTM is running, execute the Python script &#34;Single_Trial_Acquisition.py&#34; in the project repository. Follow the instructions provided by the script to capture a single trial of the participant grasping the stimulus object. 
	NOTE: The script will produce an auditory cue. This will signal to the participant to open their eyes and execute the grasp. In our demonstrations, the task is to reach and grasp the target object, lift it vertically by approximately 10 cm, set it down, and return the hand to its starting position.</li>
</ol>
9. Labeling the markers
<ol>
	<li>Within the QTM, drag and drop the individual marker trajectories from Unidentified trajectories to Labeled trajectories, and label them according to the naming convention in Figure 2.</li>
	<li>Select all the markers attached to the hand, and right-click on and select Generate AIM model from selection. In the new window, select Create new model based on Marker connections from existing AIM model and press the Next button.</li>
	<li>Select the RH_FH model definition, and press Open. Press Next, enter a name for the AIM model, and press OK. Finally, press Finish to create an AIM model for the participant&#39;s hand, which will be used to automatically identify markers in successive trials from the same participant.</li>
</ol>
10. Creating a personalized skeleton definition for the participant
<ol>
	<li>In the QTM, navigate to the Play menu, and select Play with Real-Time Output.</li>
	<li>Open Maya. Navigate to the QTM Connect shelf, and press the Connect to QTM icon. In the new window, check Markers, and press Connect. Now, press the Play icon in the QTM Connect shelf.</li>
	<li>Shift-select all the hand markers and press the Wash Locators icon. Select the washed hand markers, and press Ctrl + G. This will create a group node. Name the group Markers.</li>
	<li>Select all the hand markers. In the Modify menu, click Search and Replace Names. Search for the RH_ prefix, and delete the prefix for the markers.</li>
	<li>Press the Import Solver icon in the QTM Connect shelf. Load the skeleton definition &#34;RH_FH.xml&#34;.</li>
	<li>In the Windows menu, navigate to General Editors | Namespace Editor. Within the new window, click on :(root), and press New to create a new namespace, RH. Click on the RH namespace, press New, and name the new namespace ModelPose.</li>
	<li>Select all the markers, click on the RH namespace, and press Add Selected to add the markers to the RH namespace.</li>
	<li>Select the skeleton bones, click on the ModelPose namespace, and press Add Selected to add the skeleton bones to the ModelPose namespace.</li>
	<li>Rotate, translate, and scale the skeleton to fit the marker data. Next, for each skeleton joint individually, Shift + Select the skeleton joint and its associated markers, and press the Add Attachments icon. Finally, press the Export Solver icon to export the new skeleton definition to an XML file that can be loaded in the QTM (see next step). 
	&#8203;NOTE: This step is not strictly necessary, but it is useful to increase the accuracy of the skeleton fit to the marker data. Read the QSolverQuickstartGuide on&#160;https://github.com/qualisys/QTM-Connect-For-Maya for more information.</li>
</ol>
11. Reconstruct the joint skeletal joint poses
<ol>
	<li>Within the QTM, open the project settings by pressing the gearwheel icon. In the sidebar, navigate to Skeleton Solver, and press Load to select a skeleton definition file. Adjust the Scale Factor to 100%, and press Apply.</li>
	<li>Navigate to TSV Export, and check the boxes 3D, 6D, and Skeleton in the Data Type settings. Check all the boxes in the General settings. Press Apply, and close the project settings.</li>
	<li>Press the Reprocess icon, check the boxes Solve Skeletons and Export to TSV file, and press OK.</li>
</ol>
12. Generating hand mesh reconstructions
<ol>
	<li>Open a command window in the project repository, and activate the conda environment by executing the command: 
	conda activate contact-regions</li>
	<li>Then, execute the following command, and follow the instructions provided by the script to generate, for each frame of the trial, a hand mesh reconstructing the current hand pose. 
	python Hand_Mesh_Reconstruction.py --gpu 0 --test_epoch 4 
	&#8203;NOTE: These mesh reconstructions are generated automatically using a modified version of the open-source and pretrained hand mesh generation tool, DeepHandMesh29.</li>
</ol>
13. Generating hand-object contact region estimates
<ol>
	<li>Open a command window in the project repository, execute the following command, and follow the instructions provided by the script to generate hand and object contact region estimates by computing the intersection between the hand and object meshes. 
	blender --background --python &#34;Contact_Region_Estimation.py&#34;</li>
</ol>

The authors declare that no competing interests exist.

We propose a method that enables the estimation of contact regions for hand-object interactions during multi-digit grasps. As full tracking of the whole surface of a hand is currently intractable, we propose using a reconstruction of a hand mesh whose pose is determined by sparse key points on the hand. To track these sparse key points, our solution employs a research-grade motion capture system based on passive marker tracking. Of course, other motion capture systems could also be employed with the proposed method, granted that they yield sufficiently accurate 3D position data. We advise against active marker motion capture systems (such as the popular but discontinued Optotrak Certus), since these require attaching cables and/or electronic devices to the participants&#39; hands, which may restrict movements or at least yield less typical grasps as participants are made more consciously aware of the pose of their hands. Motion-tracking gloves using inertial measurement units may be a possibility, even though these systems are known to suffer from drift, may also restrict hand movements, and do not allow for the surface of the hand to come into full and direct contact with the object surfaces. Commercial markerless hand-tracking solutions (e.g., the Leap Motion46,47,48) may also be a possibility, although it may not be possible to track object positions with these systems alone. The most promising alternative option to a research-grade motion capture system is given by open-source, markerless tracking solutions (e.g., Mathis et al.28). If used with multiple co-registered cameras49, such systems could potentially track hand joint positions and object positions in 3D without the need of markers, gloves, or cables. These solutions, as well as this marker-based system, may suffer, however, from data loss issues due to occlusions.
Limitations and future directions 
As the hand reconstructions obtained through this method will not be fully accurate, there are some limitations to the types of experiments for which the method should be used. Deviations in hand mesh reconstructions from ground truth will manifest themselves in deviations in the estimated hand/object contact regions. Thus, applying this method to derive absolute measures would require assessing the fidelity of the contact region estimates. However, even approximate estimates can still be useful in within-participant experimental designs because the potential biases of the method are likely to affect different experimental conditions within a participant in a similar way. Therefore, statistical analyses and inferences should be performed only on measures such as the differences in contact area between conditions, where the direction of an effect will correlate with the respective ground truth. In future research, we plan to further validate our approach, for example by comparing contact region estimates to thermal fingerprints on objects covered in thermochromic paint.
Most processing steps from the data collection to the final contact region estimation are fully automated and, thus, offer important contributions toward a standardized procedure for hand-object contact region estimation. However, an initial fit of the individualized skeletons to the 3D positions of the tracked markers must still be performed manually to obtain a skeleton definition for each participant. As the number of participants for an experiment increases, so does the number of manual fits, and this is currently the most time-consuming step in the procedure and requires some familiarity with manual rigging in the Autodesk Maya Software. In the future, we aim to automate this step to avoid human influence on the procedure by adding an automatic skeleton calibration procedure.
The workflow described here relies on the Qualisys hardware and software (e.g., the QTM skeleton solver). This currently limits the accessibility of our method to laboratories that have a similar setup. In principle, however, the method can be applied to any source of motion capture data. To expand the accessibility, in ongoing work, we are exploring alternatives that should generalize our workflow and make it less reliant on specific hardware and software licenses.
Another important limitation of the method is that, in its current form, it can only be applied to rigid (non-deformable) objects. In the future, this limitation could be overcome using methods for recording the surface shape of the grasped object as it deforms. Additionally, due to its approximate nature, the method is not currently well suited to very small or thin objects.
In conclusion, by integrating state-of-the-art motion tracking with high-fidelity hand surface modeling, we provide a method to estimate hand-object contact regions during grasping and manipulation. In future research, we plan to deploy this method to investigate and model visually guided grasping behavior in humans16. We further plan to integrate these tools with eye tracking46, 50,51,52 and virtual/augmented reality systems53,54,55 to investigate visually guided hand and eye movement motor control in real and virtual naturalistic environments18,46,56,57. For these reasons, the proposed method could be of interest to researchers studying haptic perception58, motor control, and human-computer interaction in virtual and augmented reality. Finally, accurate measurements of human grasping abilities could inform the design of robust robotic systems based on the principles of interactive perception39,40,41,42,43 and may have translational applications for upper limb prosthetics.

The capacity to grasp and manipulate objects is a key ability that allows humans to reshape the environment to their wants and needs. However, controlling multi-jointed hands effectively is a challenging task that requires a sophisticated control system. This motor control system is guided by several forms of sensory input, amongst which vision is paramount. Through vision, individuals can identify the objects in the environment and estimate their position and physical properties and can then reach, grasp, and manipulate those objects with ease. Understanding the complex system that links the input at the retinae with the motor commands that control the hands is a key challenge of sensorimotor neuroscience. To model, predict, and understand how this system works, we must first be able to study it in detail. This requires high-fidelity measurements of both visual inputs and hand motor outputs.
Past motion-tracking technology has imposed a number of limitations on the study of human grasping. For example, systems requiring cables attached to the participants&#39; hands1,2 tend to restrict the range of finger motions, potentially altering the grasping movements or the measurements themselves. Despite such limitations, previous research has been able to identify several factors that influence visually guided grasping. Some of these factors include object shape3,4,5,6, surface roughness7,8,9, or the orientation of an object relative to the hand4,8,10. However, to overcome previous technological limitations, the majority of this prior research has employed simple stimuli and highly constrained tasks, thus predominantly focusing on individual factors3,4,6,7,10, two-digit precision grips3,4,6,9,11,12,13,14,15,16,17,18, single objects19, or very simple 2D shapes20,21. How previous findings generalize beyond such reduced and artificial lab conditions is unknown. Additionally, the measurement of hand-object contact is often reduced to the estimation of digit contact points22. This simplification may be appropriate to describe a small subset of grasps in which only the fingertips are in contact with an object. However, in the majority of real-world grasps, extensive regions of the fingers and palm come in contact with an object. Further, a recent study23 has demonstrated, using a haptic glove, that objects can be recognized by how their surface impinges on the hand. This highlights the importance of studying the extended contact regions between the hands and the objects grasped, not just the contact points between the objects and the fingertips22.
Recent advances in motion capture and 3D hand modeling have allowed us to move past previous limitations and to study grasping in its full complexity. Passive marker-based motion tracking is now available with millimeter-sized markers that can be attached to the back of the participant&#39;s hands to track joint movements24. Further, automatic marker identification algorithms for passive marker systems are now sufficiently robust to almost eliminate the need for the extensive manual postprocessing of marker data25,26,27. Markerless solutions are also reaching impressive levels of performance at tracking animal body parts in videos28. These motion-tracking methods, thus, finally allow reliable and non-invasive measurements of complex multi-digit hand movements24. Such measurements can inform us about joint kinematics and enable us to estimate the contact points between the hand and an object. Additionally, in recent years, the computer vision community has been tackling the problem of constructing models of the human hands that can replicate the soft tissue deformations during object grasping and even during self-contact between hand parts29,30,31,32. Such 3D mesh reconstructions can be derived from different types of data, such as video footage33,34, skeletal joints (derived from marker-based35 or markerless tracking36), and depth images37. The first key advance in this domain was provided by Romero et al.38, who derived a parametric hand model (MANO) from over 1,000 hand scans from 31 subjects in various poses. The model contains parameters for both the pose and shape of the hand, facilitating regression from different sources of data to a full hand reconstruction. The more recent DeepHandMesh29 solution builds on this approach by constructing a parametrized model through deep learning and by adding penetration avoidance, which more accurately replicates physical interactions between hand parts. By combining such hand mesh reconstructions with 3D tracked object meshes, it is, thus, now possible to estimate contact regions not just on the surface of objects32 but also on the surface of the hand.
Here, we propose a workflow that brings together the high-fidelity 3D tracking of objects and hand joints with novel hand mesh reconstruction algorithms. The method yields detailed maps of hand-object contact surfaces. These measurements will assist sensorimotor neuroscientists in extending our theoretical understanding of human visually guided grasping. Further, the method could be useful to researchers in adjacent fields. For example, human factor researchers may use this method to construct better human-machine interface systems in virtual and augmented reality18. High-fidelity measurements of human grasping behaviors can also assist roboticists in designing human-inspired robotic grasping systems based on the principles of interactive perception39,40,41,42,43. We, thus, hope that this method will help advance grasping research across neuroscience and engineering fields from sparse descriptions of highly constrained tasks to fuller characterizations of naturalistic grasping behaviors with complex objects and real-world tasks. The overall approach is outlined in Figure 1.
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/64877/64877fig01.jpg" /> 
Figure 1: Key steps in the proposed method. (A) Motion capture cameras image a workbench from multiple angles. (B) A stimulus object is 3D printed from a triangulated mesh model. (C) Four spherical reflective markers are glued to the surface of the real object. A semi-automated procedure identifies four corresponding points on the surface of the mesh object. This correspondence allows us to roto-translate the mesh model to the 3D tracked position of the real object. (D) Reflective markers are attached to different landmarks on the back of a participant&#39;s hand using double-sided tape. (E) The motion capture system acquires the trajectories in 3D space of the tracked object and hand markers during a single trial. (F) A participant-specific hand skeleton is constructed using 3D computer graphics software. Skeletal joint poses are then estimated for each frame of each trial in an experiment through inverse kinematics. (G) Joint poses are input to a modified version of DeepHandMesh29, which outputs an estimated 3D hand mesh in the current 3D pose and position. (H) Finally, we use mesh intersection to compute the hand-object contact regions. <a href="https://www.jove.com/files/ftp_upload/64877/64877fig01large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>

<table><tbody><tr><td>Anaconda Python distribution&amp;nbsp;</td><td></td><td>(Anaconda 5.3.1 or later); https://repo.anaconda.com/archive/</td><td>scripts and functions were generated in Python version 3.7</td></tr><tr><td>Autodesk Maya</td><td>Autodesk, Inc.</td><td>Maya2022; https://www.autodesk.com/products/maya/overview</td><td>3D computer graphics application.&amp;nbsp;&amp;nbsp;</td></tr><tr><td>Blender&amp;nbsp;</td><td>Blender Foundation&amp;nbsp;</td><td>Blender 2.92; https://download.blender.org/release/</td><td>3D computer graphics application.&amp;nbsp;&amp;nbsp;</td></tr><tr><td>Computer Workstation</td><td>N/A</td><td>N/A</td><td>OS: Windows 10 or higher.</td></tr><tr><td>DeepHandMesh</td><td>Meta Platforms, Inc. (Meta Open Source)</td><td>https://github.com/facebookresearch/DeepHandMesh</td><td>Pre-trained hand mesh generation tool.</td></tr><tr><td>Miqus M5</td><td>Qualisys Ab</td><td>https://www.qualisys.com/cameras/miqus/&amp;nbsp;</td><td>Passive marker motion tracking camera (8 units).</td></tr><tr><td>Miqus video camera</td><td>Qualisys Ab</td><td>https://www.qualisys.com/cameras/miqus-video/</td><td>Color video camera, synchronized with Miquis M5 tracking cameras (6 units).</td></tr><tr><td>Project repository&amp;nbsp;</td><td>N/A</td><td>Data and Code Repository&amp;nbsp;</td><td>Data and code to replicate the current project. The repository is currently under construction, but we provide a private link where reviewers can download the current and most up-to-date version of the repository. The final repository will be made publicly available upon acceptance.&amp;nbsp;</td></tr><tr><td>Python 3</td><td>Python Software Foundation</td><td>Python Version 3.7</td><td>Python3 and associated built-in libraries.</td></tr><tr><td>QTM Connect for Maya</td><td>Qualisys Ab</td><td>https://github.com/qualisys/QTM-Connect-For-Maya&amp;nbsp;</td><td>Stream skeleton, rigid bodies and marker data from QTM to Maya</td></tr><tr><td>QTM Qualisys Track Manager</td><td>Qualisys Ab</td><td>Qualisys Track Manager 2021.2; https://www.qualisys.com/software/qualisys-track-manager/&amp;nbsp;</td><td>Motion capture software</td></tr><tr><td>Qualisys SDK for Python</td><td>Qualisys Ab</td><td>https://github.com/qualisys/qualisys_python_sdk&amp;nbsp;</td><td>Implements communication between QTM and Python</td></tr></tbody></table>

estimation of contact regions between hands and objects during human multi-digit grasping

To grasp an object successfully, we must select appropriate contact regions for our hands on the surface of the object. However, identifying such regions is challenging. This paper describes a workflow to estimate the contact regions from marker-based tracking data. Participants grasp real objects, while we track the 3D position of both the objects and the hand, including the fingers' joints. We first determine the joint Euler angles from a selection of tracked markers positioned on the back of the hand. Then, we use state-of-the-art hand mesh reconstruction algorithms to generate a mesh model of the participant's hand in the current pose and the 3D position. 
Using objects that were either 3D printed or 3D scanned-and are, thus, available as both real objects and mesh data-allows the hand and object meshes to be co-registered. In turn, this allows the estimation of approximate contact regions by calculating the intersections between the hand mesh and the co-registered 3D object mesh. The method may be used to estimate where and how humans grasp objects under a variety of conditions. Therefore, the method could be of interest to researchers studying visual and haptic perception, motor control, human-computer interaction in virtual and augmented reality, and robotics.

The first requirement for the proposed method is a system to accurately track the position of 3D objects and hands. The specific setup is shown in Figure 1A and uses hardware and software produced by the motion capture company Qualisys. We position a workbench within a tracking volume (100 cm x 100 cm x 100 cm), which is imaged from multiple angles by eight tracking cameras and six video cameras arranged on a cubical frame surrounding the workspace. The tracking cameras track the 3D position of the reflective markers within the tracking volume at 180 frames/s and with sub-millimeter 3D spatial resolution. We employ 4 mm reflective markers, which are attached to the objects and hands using skin-friendly double-sided adhesive tape. The 3D marker positions are processed by the motion capture software. The discussion section also reviews alternative motion capture systems that could be employed with the proposed method.
To obtain accurate 3D reconstructions of real objects being grasped and manipulated, we propose two options. The first, which is the one adopted here, is to start from a virtual 3D object model in the form of a polygon mesh. Such 3D models can be constructed using appropriate software (e.g., Blender 3D44) and then 3D printed (Figure 1B). The second option is to take an existing, real 3D object and use 3D scanning technology to construct a mesh model replica of the object. Whichever the strategy, the end goal is to obtain both a real 3D object and the corresponding virtual 3D object mesh model. Of note, the approach described here works only with rigid (i.e., non-deformable) objects.
Once the 3D surface of an object is available as a mesh model, its position must be tracked and co-registered (Figure 1C). To do so, four non-planar reflective markers are attached to the surface of the real object, and the object is placed within the tracking volume. The 3D positions of the object markers are then briefly captured. This capture is used to establish the correspondence between the four markers and four vertices of the object mesh model. This is done using a simple ad hoc software route written in Blender&#8217;s Python API. Within Blender&#8217;s Viewport, the program presents the virtual object together with the marker positions which are represented as a single mesh object comprised of one sphere for each marker. The user can then rotate and translate the object and/or the markers to align them such that they co-align with the real markers placed on the real object. The program will register the rotations and translation that are applied to calculate a single roto-translation that is finally applied to the original object mesh, providing an object mesh that is co-registered with the rigid body definition in QTM.
Having established correspondence, whenever the real object is moved within the tracking volume, the virtual object can be placed in the new position by computing the roto-translation between the tracked markers and the four corresponding mesh vertices. To record the dynamics of the grasp instead, a total of 24 spherical reflective markers are attached on different landmarks of the hand using double-sided tape (Figure 1D and Figure 2).
At the beginning of a trial (Figure 1E), a participant places their hand flat on the workbench with the palm facing downward and closes their eyes. The experimenter places a target object on the workbench in front of the participant. Next, an auditory cue signals to the participant to open their eyes and execute the grasp. In our demonstrations, the task is to reach and grasp the target object, lift it vertically by approximately 10 cm, set it down, and return the hand to its starting position. A script written in Python 3.7 controls the experiment. On each trial, the script selects and communicates the current condition settings to the experimenter (e.g., object identity and positioning). The script also controls the trial timing, including auditory cues and the start and stop of the motion capture recordings.
Limbs are not only characterized by their position in 3D space but also by their pose. Thus, to obtain a complete 3D reconstruction of a human hand executing a real grasp, we require not only the positions of each joint in 3D space but also the relative pose (translation and rotation) of each joint with respect to its parent joint (Figure 1F). Skeletal joint positions and orientations can be inferred from marker positions using inverse kinematics. To do so, here we employ the skeleton solver provided by the QTM software. For the solver to work, we must first provide a skeleton definition that links the position and orientation of each joint to multiple marker positions. A skeleton definition is, thus, constructed, and the skeleton rig is linked to the marker data using the QTM Connect plugin for Maya. We create personalized skeleton definitions for each participant to maximize the accuracy of the skeleton fits to the marker data. For each participant, we manually fit a hand skeleton to a single frame of motion capture data. Having obtained a participant-specific skeleton definition, we then run the skeleton solver to estimate the skeletal joint poses for each frame of each trial in the experiment.
For each frame of each trial in an experiment, we generate a hand mesh that reconstructs the current hand pose using the open-source and pretrained hand mesh generation tool, DeepHandMesh28 (Figure 1G). DeepHandMesh is a deep encoder-decoder network that generates personalized hand meshes from images. First, the encoder estimates the pose of a hand within an image (i.e., the joint Euler angles). Then, the estimated hand pose and a personalized ID vector are input to the decoder, which estimates a set of three additive correctives to a rigged template mesh. Finally, the template mesh is deformed according to the estimated hand pose and correctives using linear blend skinning. The first corrective is an ID-dependent skeleton corrective through which the skeletal rig is adjusted to incorporate the person-specific joint positions. The other two correctives are mesh correctives through which the mesh vertices are adjusted to better represent the hand surface of the participant. One of the mesh correctives is an ID-dependent mesh corrective that accounts for the surface structure of an individual participant&#39;s hand. The final mesh corrective instead is a pose-dependent vertex corrective that accounts for hand surface deformations due to the current hand pose.
DeepHandMesh is trained using weak supervision with 2D joint key points and scene depth maps. Here, we use only the pretrained DeepHandMesh decoder to generate hand mesh reconstructions, modified in the following ways (Figure 3). First, as the network is not trained on specific participants, the generic ID-dependent mesh corrective provided with the pretrained model is employed (Figure 3A). Further, the ID-dependent skeleton corrective is derived using the QTM skeleton solver as described above (Figure 3B). Proportional scaling of the hand with the skeleton length is assumed, and the mesh thickness is uniformly scaled by a factor derived from the relative scaling of the skeleton such that the mesh better approximates the participant&#39;s hand size (Figure 3C). This modified mesh is input to the decoder, together with the current hand pose (derived from the marker data) and the 3D position and orientation of the wrist. The decoder, thus, computes the current pose-dependent corrective, applies all the correctives and roto-translations, and outputs a 3D hand mesh reconstruction of the current hand pose in the same coordinate frame as the 3D tracked object mesh (Figure 3D).
<img alt="Figure 3" class="xfigimg" src="/files/ftp_upload/64877/64877fig03.jpg" /> 
Figure 3: Modifications to the pretrained DeepHandMesh decoder. (A) Fixed, generic ID-dependent mesh corrective. (B) ID-dependent skeleton corrective derived through inverse kinematics in step 10. (C) The size of the hand mesh is scaled by the same factor as the skeletal joints. (D) Final 3D hand mesh reconstruction of the current hand pose. <a href="https://www.jove.com/files/ftp_upload/64877/64877fig03large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
Having reconstructed 3D mesh models for both a participant&#39;s hand and a grasped object, hand-object contact regions can be estimated by computing the intersection between the hand and object meshes (Figure 1H). The assumption behind this is that the real hand is deformed by contact with the surface, meaning the skeleton can come closer to the surface than would be possible if the hand were rigid, which allows portions of the hand mesh to pass through the object mesh. As a result, the contact areas can be approximated as the regions of overlap between the two meshes.
Specifically, to compute these regions of overlap, we define object mesh vertices that are contained within the 3D volume of the hand mesh as being in contact with the hand. These vertices are identified using a standard raytracing approach45. For each vertex of the object mesh, a ray is cast from that vertex to an arbitrary 3D point outside the hand mesh. We then assess the number of intersections that occur between the cast ray and the triangles composing the hand&#39;s surface. If the number of intersections is odd, the object vertex is contained inside the hand mesh. If the number of intersections is even, then the object vertex is outside the hand mesh. The contact regions on the surface of the object can, thus, be approximated as the set of triangle faces whose vertices are all contained within the hand mesh. We can apply the same rationale to the hand mesh vertices contained in the 3D volume of the object mesh to estimate the contact regions on the surface of the hand. Notably, more advanced approaches to Boolean mesh operations could also be used31.
Video 1 shows a video of a hand, tracked points, and co-registered mesh all moving side-by-side during a single grasp to a 3D-printed cat figurine. Figure 4A instead shows a single frame at the time of hand-object contact from a grasp to a 3D-printed croissant, together with the hand-object mesh reconstructions (Figure 4B) and the estimated contact regions on the surface of the croissant (Figure 4C).
<img alt="Figure 4" class="xfigimg" src="/files/ftp_upload/64877/64877fig04.jpg" /> 
Figure 4: Estimated hand-object contact regions. (A) Tracked hand and object viewed from one of the tracking cameras during a grasp. (B) Reconstructed hand mesh and tracked object mesh rendered from the same viewpoint as the tracking camera. (C) Contact regions on the surface of the object seen from multiple viewpoints. <a href="https://www.jove.com/files/ftp_upload/64877/64877fig04large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
Video 1: Mesh reconstructions of the hand and object. Gif animation of the hand, tracked markers, and the hand and object mesh reconstructions during a single grasp viewed from the same camera viewpoint. <a href="https://www.jove.com/files/ftp_upload/64877/Movie01.gif" target="_blank">Please click here to download this Video.</a>

Watch this Scientific Journal Video about Estimation of Contact Regions Between Hands and Objects During Human Multi-Digit Grasping at JoVE.com

Estimation of Contact Regions Between Hands and Objects During Human Multi-Digit Grasping

When we grasp an object, multiple regions of the fingers and hand typically make contact with the object's surface. Reconstructing such contact regions is challenging. Here, we present a method for approximately estimating the contact regions by combining marker-based motion capture with existing deep learning-based hand mesh reconstruction.

To grasp an object successfully, we must select appropriate contact regions for our hands on the surface of the object. However, identifying such regions ...

estimation-contact-regions-between-hands-objects-during-human-multi

Research

JoVE Journal

Behavior

1.5K Views.  Justus Liebig University Giessen. Eerder gedragsonderzoek naar menselijk grijpen is beperkt gebleven tot zeer beperkte metingen in strak gecontroleerde scenario's. Ons protocol maakt een veel rijkere karakterisering van complex naturalistisch grijpgedrag mogelijk. Deze techniek maakt gebruik van gedetailleerde kaarten van contactoppervlakken van handobjecten als gevolg van meercijferige grepen.Dit stelt ons in staat om te onderzoeken hoe mensen objecten met een ongekend niveau van verfijning grijpen. Nauwkeurige meting...

Video: Estimation of Contact Regions Between Hands and Objects During Human Multi-Digit Grasping

One Dimensional Turing-Like Handshake Test for Motor Intelligence

In the Turing test, a computer model is deemed to "think intelligently" if it can generate answers that are not distinguishable from those of a human. However, this test is limited to the linguistic aspects of machine intelligence. A salient function of the brain is the control of movement, and the movement of the human hand is a sophisticated demonstration of this function. Therefore, we propose a Turing-like handshake test, for machine motor intelligence. We administer the test through a telerobotic system in which the interrogator is engaged in a task of holding a robotic stylus and interacting with another party (human or artificial). Instead of asking the interrogator whether the other party is a person or a computer program, we employ a two-alternative forced choice method and ask which of two systems is more human-like. We extract a quantitative grade for each model according to its resemblance to the human handshake motion and name it "Model Human-Likeness Grade" (MHLG). We present three methods to estimate the MHLG. (i) By calculating the proportion of subjects' answers that the model is more human-like than the human; (ii) By comparing two weighted sums of human and model handshakes we fit a psychometric curve and extract the point of subjective equality (PSE); (iii) By comparing a given model with a weighted sum of human and random signal, we fit a psychometric curve to the answers of the interrogator and extract the PSE for the weight of the human in the weighted sum. Altogether, we provide a protocol to test computational models of the human handshake. We believe that building a model is a necessary step in understanding any phenomenon and, in this case, in understanding the neural mechanisms responsible for the generation of the human handshake.

We present a Turing-like Handshake test administered through a telerobotic system in which the interrogator is holding a robotic stylus and interacting with another party (human or artificial). We use a forced choice method, and extract a measure for the similarity of the artificial model to a human handshake.

In the Turing test, a computer model is deemed to "think intelligently" if it can generate answers that are not distinguishable from those of a human.  ...

Neuroscience

Testing Tactile Masking between the Forearms

Masking, in which one stimulus affects the detection of another, is a classic technique that has been used in visual, auditory, and tactile research, usually using stimuli that are close together to reveal local interactions. Masking effects have also been demonstrated in which a tactile stimulus alters the perception of a touch at a distant location. Such effects can provide insight into how components of the body's representations in the brain may be linked. Occasional reports have indicated that touches on one hand or forearm can affect tactile sensitivity at corresponding contralateral locations. To explore the matching of corresponding points across the body, we can measure the spatial tuning and effect of posture on contralateral masking. Careful controls are required to rule out direct effects of the remote stimulus, for example by mechanical transmission, and also attention effects in which thresholds may be altered by the participant's attention being drawn away from the stimulus of interest. The use of this technique is beneficial as a behavioural measure for exploring which parts of the body are functionally connected and whether the two sides of the body interact in a somatotopic representation. This manuscript describes a behavioural protocol that can be used for studying contralateral tactile masking.

Here we explore contralateral tactile masking between the forearms in which tactile detection thresholds are modulated by vibration applied to a remote site. The details of which remote sites have an effect can tell us about how the body is represented in the brain.

Masking, in which one stimulus affects the detection of another, is a classic technique that has been used in visual, auditory, and tactile research, ...

Behavioral Assessment of Manual Dexterity in Non-Human Primates

The corticospinal (CS) tract is the anatomical support of the exquisite motor ability to skillfully manipulate small objects, a prerogative mainly of primates1. In case of lesion affecting the CS projection system at its origin (lesion of motor cortical areas) or along its trajectory (cervical cord lesion), there is a dramatic loss of manual dexterity (hand paralysis), as seen in some tetraplegic or hemiplegic patients. Although there is some spontaneous functional recovery after such lesion, it remains very limited in the adult. Various therapeutic strategies are presently proposed (e.g. cell therapy, neutralization of inhibitory axonal growth molecules, application of growth factors, etc), which are mostly developed in rodents. However, before clinical application, it is often recommended to test the feasibility, efficacy, and security of the treatment in non-human primates. This is especially true when the goal is to restore manual dexterity after a lesion of the central nervous system, as the organization of the motor system of rodents is different from that of primates1,2. Macaque monkeys are illustrated here as a suitable behavioral model to quantify manual dexterity in primates, to reflect the deficits resulting from lesion of the motor cortex or cervical cord for instance, measure the extent of spontaneous functional recovery and, when a treatment is applied, evaluate how much it can enhance the functional recovery.
The behavioral assessment of manual dexterity is based on four distinct, complementary, reach and grasp manual tasks (use of precision grip to grasp pellets), requiring an initial training of adult macaque monkeys. The preparation of the animals is demonstrated, as well as the positioning with respect to the behavioral set-up. The performance of a typical monkey is illustrated for each task. The collection and analysis of relevant parameters reflecting precise hand manipulation, as well as the control of force, are explained and demonstrated with representative results. These data are placed then in a broader context, showing how the behavioral data can be exploited to investigate the impact of a spinal cord lesion or of a lesion of the motor cortex and to what extent a treatment may enhance the spontaneous functional recovery, by comparing different groups of monkeys (treated versus sham treated for instance). Advantages and limitations of the behavioral tests are discussed. The present behavioral approach is in line with previous reports emphasizing the pertinence of the non-human primate model in the context of nervous system diseases2,3.

As manual dexterity is a prerogative mainly of primates, behavioral tasks have been developed in macaque monkeys. Four reach and grasp prehension tasks, measuring hand manipulation ability and force, allow to establish functional recovery after a lesion of the central nervous system and to test the effect of a treatment.

The corticospinal (CS) tract is the anatomical support of the exquisite motor ability to skillfully manipulate small objects, a prerogative mainly of ...

Frame-by-Frame Video Analysis of Idiosyncratic Reach-to-Grasp Movements in Humans

Prehension, the act of reaching to grasp an object, is central to the human experience. We use it to feed ourselves, groom ourselves, and manipulate objects and tools in our environment. Such behaviors are impaired by many sensorimotor disorders, yet our current understanding of their neural control is far from complete. Current technologies for investigating human reach-to-grasp movements often utilize motion tracking systems that can be expensive, require the attachment of markers or sensors to the hands, impede natural movement and sensory feedback, and provide kinematic output that can be difficult to interpret. While generally effective for studying the stereotypical reach-to-grasp movements of healthy sighted adults, many of these technologies face additional limitations when attempting to study the unpredictable and idiosyncratic reach-to-grasp movements of young infants, unsighted adults, and patients with neurological disorders. Thus, we present a novel, inexpensive, and highly reliable yet flexible protocol for quantifying the temporal and kinematic structure of idiosyncratic reach-to-grasp movements in humans. High speed video cameras capture multiple views of the reach-to-grasp movement. Frame-by-frame video analysis is then used to document the timing and magnitude of pre-defined behavioral events such as movement start, collection, maximum height, peak aperture, first contact, and final grasp. The temporal structure of the movement is reconstructed by documenting the relative frame number of each event while the kinematic structure of the hand is quantified using the ruler or measure function in photo editing software to calibrate 2 dimensional linear distances between two body parts or between a body part and the target. Frame-by-frame video analysis can provide a quantitative and comprehensive description of idiosyncratic reach-to-grasp movements and will enable researchers to expand their area of investigation to include a greater range of naturalistic prehensile behaviors, guided by a wider variety of sensory modalities, in both healthy and clinical populations.

This protocol describes how to use frame-by-frame video analysis to quantify idiosyncratic reach-to-grasp movements in humans. A comparative analysis of reaching in sighted versus unsighted healthy adults is used to demonstrate the technique, but the method can also be applied to the study of developmental and clinical populations.

Prehension, the act of reaching to grasp an object, is central to the human experience. We use it to feed ourselves, groom ourselves, and manipulate ...

Measurement of Spatial Stability in Precision Grip

The purpose of the protocol is to indirectly evaluate the direction of the finger force during manipulation of a handheld object based on the biomechanical relationships in which deviated force direction causes center of pressure (COP) replacement. To evaluate this, a thin, flexible, and high spatial resolution pressure sensor sheet is used. The system allows measurement of the COP trajectory in addition to the force amplitude and its temporal regulation. A series of experiments found that increased trajectory length reflected a sensorimotor deficit in stroke patients, and that decreased COP trajectory reflects a compensatory strategy to avoid an object slipping from the hand grip in the elderly. Moreover, the COP trajectory could also be decreased by dual task interference. This article describes the experimental procedure and discusses how finger COP contributes to an understanding of the physiology and pathophysiology of grasping.

The goal of this protocol is to measure the center of pressure (COP) replacement using a high spatial resolution sensor sheet to reflect the spatial stability in a precision grip. The use of this protocol could contribute to greater understanding of the physiology and pathophysiology of grasping.

The purpose of the protocol is to indirectly evaluate the direction of the finger force during manipulation of a handheld object based on the ...

Design and Use of an Apparatus for Presenting Graspable Objects in 3D Workspace

Reaching and grasping are highly-coupled movements, and their underlying neural dynamics have been widely studied in the last decade. To distinguish reaching and grasping encodings, it is essential to present different object identities independent of their positions. Presented here is the design of an automatic apparatus that is assembled with a turning table and three-dimensional (3D) translational device to achieve this goal. The turning table switches different objects corresponding to different grip types while the 3D translational device transports the turning table in 3D space. Both are driven independently by motors so that the target position and object are combined arbitrarily. Meanwhile, wrist trajectory and grip types are recorded via the motion capture system and touch sensors, respectively. Furthermore, representative results that demonstrate successfully trained monkey using this system are described. It is expected that this apparatus will facilitate researchers to study kinematics, neural principles, and brain-machine interfaces related to upper limb function.

Presented here is a protocol to build an automatic apparatus that guides a monkey to perform the flexible reach-to-grasp task. The apparatus combines a 3D translational device and turning table to present multiple objects in an arbitrary position in 3D space.

Reaching and grasping are highly-coupled movements, and their underlying neural dynamics have been widely studied in the last decade. To distinguish ...

Tactile Semiautomatic Passive-Finger Angle Stimulator (TSPAS)

Passive tactile perception is the ability to passively and statically perceive stimulus information coming from the skin; for example, the ability to sense spatial information is the strongest in the skin on the hands. This ability is termed tactile spatial acuity, and is measured by the tactile threshold or discrimination threshold. At present, the two-point threshold is extensively used as a measure of tactile spatial acuity, although many studies have indicated that critical deficits exist in two-point discrimination. Therefore, a computer-controlled tactile stimulus system was developed, the tactile semiautomated passive-finger angle stimulator (TSPAS), using the tactile angle discrimination threshold as a new measure for tactile spatial acuity. The TSPAS is a simple, easily operated system that applies raised angle stimuli to a subject&#39;s passive fingerpad, while controlling movement speed, distance, and contact duration. The components of the TSPAS are described in detail as well as the procedure to calculate the tactile angle discrimination threshold.

Tactile Semiautomatic Passive-Finger Angle Stimulator TSPAS

Presented is the tactile semiautomated passive-finger angle stimulator TSPAS, a new way to assess tactile spatial acuity and tactile angle discrimination using a computer-controlled tactile stimulus system that applies raised angle stimuli to a subject&#39;s passive fingerpad, while controlling for movement speed, distance, and contact duration.

Passive tactile perception is the ability to passively and statically perceive stimulus information coming from the skin; for example, the ability to ...

Measurement of the Hand Transmitted Vibration of the Human Hand Arm System During Operation of a Hand Tractor

Operators of hand tractors are exposed to high levels of hand transmitted vibration (HTV). This vibration, which can be both irksome and hazardous to human health, is imparted to the operator via his or her hands and arms. However, a standardized method for measuring HTV of hand tractors has yet to be defined. The aim of the study was to present an experimental method for the investigation of the biodynamic response and vibration transmissibility of the hand-arm system during the operation of a hand tractor in a stationary mode. Measurements were performed with ten subjects using three grip forces and three handle vibration levels to examine the influences of the hand pressure and frequency on hand transmitted vibration (HTV). The results indicate that the tightness of grip on the handle influences the vibration response of the hand-arm system, especially at frequencies between 20 and 100 Hz. The transmission of lower frequencies in the hand-arm system was relatively unattenuated. In comparison, attenuation was found to be quite marked for higher frequencies during the operation of the hand tractor. The vibration transmissibility to different parts of the hand-arm system decreased with the increase of the distance from the vibration source. The proposed methodology contributes to the collection of consistent data for the evaluation of operator vibration exposure and the ergonomics development of hand tractors.

Here, we present a standardized method for measurement of the hand transmitted vibration from handles of a single-axle tractor with special reference to changes in grip force and vibration frequency.

Operators of hand tractors are exposed to high levels of hand transmitted vibration (HTV). This vibration, which can be both irksome and hazardous to ...

Engineering

Revolutionizing Stroke Rehabilitation with Enhanced Grasping and Finger Recovery

A Flexible Wearable Supernumerary Robotic Limb for Chronic Stroke Patients

In this study, we present a flexible wearable supernumerary robotic limb that helps chronic stroke patients with finger rehabilitation and grasping movements. The design of this innovative limb draws inspiration from bending pneumatic muscles and the unique characteristics of an elephant's trunk tip. It places a strong emphasis on crucial factors such as lightweight construction, safety, compliance, waterproofing, and achieving a high output-to-weight/pressure ratio. The proposed structure enables the robotic limb to perform both envelope and fingertip grasping. Human-robot interaction is facilitated through a flexible bending sensor, detecting the wearer's finger movements and connecting them to motion control via a threshold segmentation method. Additionally, the system is portable for versatile daily use. To validate the effectiveness of this innovation, real-world experiments involving six chronic stroke patients and three healthy volunteers were conducted. The feedback received through questionnaires indicates that the designed mechanism holds immense promise in assisting chronic stroke patients with their daily grasping activities, potentially improving their quality of life and rehabilitation outcomes.

Author Spotlight: Enhancing Grasping Abilities for Hemiplegic Patients with Flexible Robotic Limbs 

This protocol introduces a flexible wearable supernumerary robotic limb tailored to assist in finger rehabilitation for stroke patients. The design incorporates a bending sensor to facilitate seamless human-robot interaction. Validation through experiments involving both healthy volunteers and stroke patients underscores the efficacy and dependability of the proposed study.

In this study, we present a flexible wearable supernumerary robotic limb that helps chronic stroke patients with finger rehabilitation and grasping ...

Capturing Dynamic Finger Gesturing with High-resolution Surface Electromyography and Computer Vision

Finger gestures are a critical element in human communication, and as such, finger gesture recognition is widely studied as a human-computer interface for state-of-the-art prosthetics and optimized rehabilitation. Surface electromyography (sEMG), in conjunction with deep learning methods, is considered a promising method in this domain. However, current methods often rely on cumbersome recording setups and the identification of static hand positions, limiting their effectiveness in real-world applications. The protocol we report here presents an advanced approach combining a wearable surface EMG and finger&#160;tracking system to capture comprehensive data during dynamic hand movements. The method records muscle activity from soft printed electrode arrays (16 electrodes) placed on the forearm as subjects perform gestures in different hand positions and during movement. Visual instructions prompt subjects to perform specific gestures while EMG and finger positions are recorded. The integration of synchronized EMG recordings and finger tracking data enables comprehensive analysis of muscle activity patterns and corresponding gestures. The reported approach demonstrates the potential of combining EMG and visual tracking technologies as an important resource for developing intuitive and responsive gesture recognition systems with applications in prosthetics, rehabilitation, and interactive technologies. This protocol aims to guide researchers and practitioners, fostering further innovation and application of gesture recognition in dynamic and real-world scenarios.

The paper presents a comprehensive protocol for simultaneously recording hand-electromyography (EMG) and visual finger tracking during natural finger gesturing. The visual data is designed to serve as the ground truth for the development of accurate EMG-based computational models for finger gesture recognition.

Finger gestures are a critical element in human communication, and as such, finger gesture recognition is widely studied as a human-computer interface for ...

Estimation of Contact Regions Between Hands and Objects During Human Multi-Digit Grasping

Samenvatting

Meer video's verkennen