Name: a psychophysics paradigm for the collection and analysis of similarity judgments
Uploaded: 2022-03-01T13:45:00
Description: Watch this Scientific Journal Video about A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments at JoVE.com

The work is supported by funding from the National Institutes of Health (NIH), grant EY07977. The authors would also like to thank Usman Ayyaz for his assistance in testing the software, and Muhammad Naeem Ayyaz for his comments on the manuscript.

Prior to beginning the experiments, all subjects provide informed consent in accordance with institutional guidelines and the Declaration of Helsinki. In the case of this study, the protocol was approved by the institutional review board of Weill Cornell Medical College.
1. Installation and set-up
<ol>
	<li>Download the code from the GitHub repository, similarities (https://github.com/jvlab/similarities). In the command line, run:&#160;git clone https://github.com/jvlab/...

<ol>
	<li>Edelman, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Representation+is+representation+of+similarities.">Representation is representation of similarities.</a> TheBehavioral and Brain Sciences. 21 (4), 449-498 (1998).</li><li>Hahn, U., Chater, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Concepts+and+similarity.">Concepts and similarity.</a> Knowledge, Concepts and Categories. , 43-84 (1997).</li><li>Kriegeskorte, N., Kievit, R. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Representational+geometry:+integrating+cognition,+computation,+and+the+brain.">Representational geometry: integrating cognition, computation, and the brain.</a> Trends in Cognitive Sciences. 17 (8), 401-412 (2013).</li><li>Hebart, M. N., Zheng, C. Y., Pereira, F., Baker, C. I. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Revealing+the+multidimensional+mental+representations+of+natural+objects+underlying+human+similarity+judgements.">Revealing the multidimensional mental representations of natural objects underlying human similarity judgements.</a> Nature Human Behaviour. 4 (11), 1173-1185 (2020).</li><li>Deng, W. S., Sloutsky, V. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+development+of+categorization:+Effects+of+classification+and+inference+training+on+category+representation.">The development of categorization: Effects of classification and inference training on category representation.</a> Developmental Psychology. 51 (3), 392-405 (2015).</li><li>Shepard, R. N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Stimulus+and+response+generalization:+tests+of+a+model+relating+generalization+to+distance+in+psychological+space.">Stimulus and response generalization: tests of a model relating generalization to distance in psychological space.</a> Journal of Experimental Psychology. 55 (6), 509-523 (1958).</li><li>Coombs, C. H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+method+for+the+study+of+interstimulus+similarity.">A method for the study of interstimulus similarity.</a> Psychometrika. 19 (3), 183-194 (1954).</li><li>Ga&#776;rdenfors, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Conceptual Spaces: The Geometry of Thought. , (2000).</li><li>Zaidi, Q., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Perceptual+spaces:+mathematical+structures+to+neural+mechanisms.">Perceptual spaces: mathematical structures to neural mechanisms.</a> The Journal of Neuroscience The Official Journal of the Society for Neuroscience. 33 (45), 17597-17602 (2013).</li><li>Krishnaiah, P. R., Kanal, L. N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Handbook of Statistics 2. , (1982).</li><li>Tsogo, L., Masson, M. H., Bardot, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Multidimensional+Scaling+Methods+for+Many-Object+Sets:+A+Review.">Multidimensional Scaling Methods for Many-Object Sets: A Review.</a> Multivariate Behavioral Research. 35 (3), 307-319 (2000).</li><li>Kriegeskorte, N., Mur, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Inverse+MDS:+Inferring+dissimilarity+structure+from+multiple+item+arrangements.">Inverse MDS: Inferring dissimilarity structure from multiple item arrangements.</a> Frontiers in Psychology. 3, 245 (2012).</li><li>Rao, V. R., Katz, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Alternative+Multidimensional+Scaling+Methods+for+Large+Stimulus+Sets.">Alternative Multidimensional Scaling Methods for Large Stimulus Sets.</a> Journal of Marketing Research. 8 (4), 488-494 (1971).</li><li>Hoffman, J. I. E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Hypergeometric+Distribution.">Hypergeometric Distribution.</a> Biostatistics for Medical and Biomedical Practitioners. , 179-182 (2015).</li><li>Victor, J. D., Rizvi, S. M., Conte, M. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Two+representations+of+a+high-dimensional+perceptual+space.">Two representations of a high-dimensional perceptual space.</a> Vision Research. 137, 1-23 (2017).</li><li>Knoblauch, K., Maloney, L. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Estimating+classification+images+with+generalized+linear+and+additive+models.">Estimating classification images with generalized linear and additive models.</a> Journal of Vision. 8 (16), 1-19 (2008).</li><li>Maloney, L. T., Yang, J. N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Maximum+likelihood+difference+scaling.">Maximum likelihood difference scaling.</a> Journal of Vision. 3 (8), 573-585 (2003).</li><li>Logvinenko, A. D., Maloney, L. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+proximity+structure+of+achromatic+surface+colors+and+the+impossibility+of+asymmetric+lightness+matching.">The proximity structure of achromatic surface colors and the impossibility of asymmetric lightness matching.</a> Perception &amp; Psychophysics. 68 (1), 76-83 (2006).</li><li>Zhou, Y., Smith, B. H., Sharpee, T. O. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Hyperbolic+geometry+of+the+olfactory+space.">Hyperbolic geometry of the olfactory space.</a> Science Advances. 4 (8), (2018).</li><li>Goldstone, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=An+efficient+method+for+obtaining+similarity+data.">An efficient method for obtaining similarity data.</a> Behavior Research Methods, Instruments, &amp; Computers. 26 (4), 381-386 (1994).</li><li>Townsend, J. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Theoretical+analysis+of+an+alphabetic+confusion+matrix.">Theoretical analysis of an alphabetic confusion matrix.</a> Perception &amp; Psychophysics. 9, 40-50 (1971).</li></ol>

The protocol outlined here is effective for obtaining and analyzing similarity judgments for stimuli that can be presented visually. The experimental paradigm, the analysis, and possible extensions are discussed first, and later the advantages and disadvantages of the method.
Experimental paradigm: The proposed method is demonstrated using a domain of 37 animal names, and a sample dataset of perceptual judgments is provided so that one can follow the analysis in step 5 and rep...

Humans mentally process and represent incoming sensory information to perform a wide range of tasks, such as object recognition, navigation, making inferences about the environment, and many others. Similarity judgments are commonly used to probe these mental representations1. Understanding the structure of mental representations can provide insight into the organization of conceptual knowledge2. It is also possible to gain insight into neural computations, by relating similarity judgments to brain activation patterns3. Additionally, similarity judgments reveal features that are salient in perception4. Studying how mental representations change during development can shed light on how they are learned5. Thus, similarity judgments provide valuable insight into information processing in the brain.
A common&#160;model of mental representations using similarities is a geometric space model6,7,8. Applied to sensory domains, this kind of model is often referred to as a perceptual space9. Points in the space represent stimuli and distances between points correspond to the perceived dissimilarity between them. From similarity judgments, one can obtain quantitative estimates of dissimilarities. These pairwise dissimilarities (or perceptual distances) can then be used to model the perceptual space via multidimensional scaling10.
There are many methods for collecting similarity judgments, each with its advantages and disadvantages. The most straightforward way of obtaining quantitative measures of dissimilarity is to ask subjects to rate on a scale the degree of dissimilarity between each pair of stimuli. While this is relatively quick, estimates tend to be unstable across long sessions as subjects cannot go back to previous judgments, and context effects, if present, cannot be detected. (Here, a context effect is defined as a change in the judged similarity between two stimuli, based on the presence of other stimuli that are not being compared.) Alternatively, subjects can be asked to compare all pairs of stimuli to all other pairs of stimuli. While this would yield a more reliable rank ordering of dissimilarities, the number of comparisons required scales with the fourth power of the number of stimuli, making it feasible for only small stimulus sets. Quicker alternatives, like sorting into a predefined number of clusters11&#160;or free sorting have their own limitations. Free sorting (into any number of piles) is intuitive, but it forces the subject to categorize the stimuli, even if the stimuli do not easily lend themselves to categorization. The more recent multi-arrangement method, inverse MDS, circumvents many of these limitations and is very efficient12. However, this method requires subjects to project their mental representations onto a 2D Euclidean plane and to consider similarities in a specific geometric manner, making the assumption that similarity structure can be recovered from Euclidean distances on a plane. Thus, there remains a need for an efficient method to collect large amounts of similarity judgments, without making assumptions about the geometry underlying the judgments.
Described here is a method that is both reasonably efficient and also avoids the above potential pitfalls. By asking subjects to rank stimuli in order of similarity to a central reference in each trial13, relative similarity can be probed directly, without assuming anything about the geometric structure of the subjects&#39; responses. The paradigm repeats a subset of comparisons with both identical and different contexts, allowing for direct assessment of context effects as well as the acquisition of graded responses in terms of choice probabilities. The analysis procedure decomposes these rank judgments into multiple pairwise comparisons and uses them to build and search for Euclidean models of perceptual spaces that explain the judgments. The method is suitable for describing in detail the representation of stimulus sets of moderate sizes (e.g., 19 to 49).
To demonstrate the feasibility of the approach, an experiment was conducted, using a set of 37 animals as stimuli. Data was collected over the course of 10 one-hour sessions and then analyzed separately for each subject. Analysis revealed consistency across subjects and negligible context effects. It also assessed consistency of perceived dissimilarities between stimuli with Euclidean models of their perceptual spaces. The paradigm and analysis procedures outlined in this paper are flexible and are expected to be of use to researchers interested in characterizing the geometric properties of a range of perceptual spaces.

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>Computer Workstation</td>
			<td>N/A</td>
			<td>N/A</td>
			<td>OS: Windows/ MacOS 10 or higher/ Linux; 3.1 GHz Dual-Core Intel Core i5 or similar; 8GB or more memory; User permissions for writing and executing files</td>
		</tr>
		<tr>
			<td>conda</td>
			<td></td>
			<td>Version 4.11</td>
			<td>OS: Windows/ MacOS 10 or higher/ Linux</td>
		</tr>
		<tr>
			<td>Microsoft Excel</td>
			<td>Microsoft</td>
			<td>Any</td>
			<td>To open and shuffle rows and columns in trial conditions files.</td>
		</tr>
		<tr>
			<td>PsychoPy</td>
			<td>N/A</td>
			<td>Version 2021.2</td>
			<td>Framework for running psychophysical studies</td>
		</tr>
		<tr>
			<td>Python 3</td>
			<td>Python Software Foundation</td>
			<td>Python Version 3.8</td>
			<td>Python3 and associated built-in libraries</td>
		</tr>
		<tr>
			<td>Required Python Libraries</td>
			<td>N/A</td>
			<td>numpy version: 1.17.2 or higher; matplotlib version 3.4.3 or higher; scipy version 1.3.1 or higher; pandas version 0.25.3 or higher; seaborn version 0.9.0 or higher; scikit_learn version 0.23.1 or higher; yaml version 6.0 or higher&#160;</td>
			<td>numpy, scipy and scikit_learn are computing modules with in-built functions for optimization and vector operations. matplotlib and seaborn are plotting libraries. pandas is used to reading in and edit data from csv files.</td>
		</tr>
	</tbody>
</table>

a psychophysics paradigm for the collection and analysis of similarity judgments

Similarity judgments are commonly used to study mental representations and their neural correlates. This approach has been used to characterize perceptual spaces in many domains: colors, objects, images, words, and sounds. Ideally, one might want to compare estimates of perceived similarity between all pairs of stimuli, but this is often impractical. For example, if one asks a subject to compare the similarity of two items with the similarity of two other items, the number of comparisons grows with the fourth power of the stimulus set size. An alternative strategy is to ask a subject to rate similarities of isolated pairs, e.g., on a Likert scale. This is much more efficient (the number of ratings grows quadratically with set size rather than quartically), but these ratings tend to be unstable and have limited resolution, and the approach also assumes that there are no context effects.
Here, a novel ranking paradigm for efficient collection of similarity judgments is presented, along with an analysis pipeline (software provided) that tests whether Euclidean distance models account for the data. Typical trials consist of eight stimuli around a central reference stimulus: the subject ranks stimuli in order of their similarity to the reference. By judicious selection of combinations of stimuli used in each trial, the approach has internal controls for consistency and context effects. The approach was validated for stimuli drawn from Euclidean spaces of up to five dimensions. 
The approach is illustrated with an experiment measuring similarities among 37 words. Each trial yields the results of 28 pairwise comparisons of the form, "Was A more similar to the reference than B was to the reference?" While directly comparing all pairs of pairs of stimuli would have required 221445 trials, this design enables reconstruction of the perceptual space from 5994 such comparisons obtained from 222 trials.

Figure 1A shows part of a conditions file generated by the script in step 3.3, for the word experiment. Each row corresponds to a trial. The stimulus in the ref column appears in the center of the display. The column names stim1 to stim8 correspond to eight positions along a circle, running counterclockwise, starting from the position to the right of the central reference. A sample trial from the word experiment is shown in Figure 1B.
<p cla...

Watch this Scientific Journal Video about A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments at JoVE.com

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments

The protocol presents an experimental psychophysics paradigm to obtain large quantities of similarity judgments, and an accompanying analysis workflow. The paradigm probes context effects and enables modeling of similarity data in terms of Euclidean spaces of at least five dimensions.

Similarity judgments are commonly used to study mental representations and their neural correlates. This approach has been used to characterize perceptual ...

This is an efficient method for collecting and analyzing similarity judgments, and it doesn't assume anything about the geometric properties of subjects underlying mental representations. The main advantages of the method are its flexibility and minimization of assumptions about the nature of the perceptual representation. The kinds of stimuli and the complexity of trials can be varied, and a wide range of geometric models can be fit to the data.This method could be useful to researchers interested in characterizing the mental representations of low and high-level aspects of visual stimuli. Select an experiment to run. Navigate to the word experiment by clicking similarities, then experiments, then word_exp, or to the image experiment by clicking similarities, then experiments, then image_exp.Finalize the experimental stimuli. If the word experiment is being run, prepare a list of words. And for the image experiment, make a new directory and place all the stimulus images in it.In the experiments directory, find the configuration file called config. yaml by clicking similarities, then experiments, and then config.yaml. Open the file in a source code editor and update the value of the file variable to the path to the directory containing the stimulus set.This is where PsychoPy will look for the image stimuli. Create trial configurations by opening the config. yaml file in the analysis directory, then set the value of the path_2_stimulus_list parameter to the path to stimuli.txt.From the similarities directory, run the script by executing the commands displayed in the window one after the other. This creates a file called trial_conditions. csv in similarities in which each row contains the names of the stimuli appearing in a trial along with their positions.Break the full set of 222 trials generated into sessions and randomize the trial order by executing the commands displayed in the window. In the typical design, sessions comprise 111 trials, each of which requires approximately one hour to run. When prompted, enter the displayed input parameters.Rename and save each of the generated files as conditions. csv in its own directory. Copy the conditions.csv file and paste it into the current directory containing the psyexp file. Open PsychoPy and open the psyexp or py file in the relevant experiments directory. In PsychoPy, click on the green play button to run the experiment.In the modal pop-up, enter the subject name or ID and session number and click OK to start. Instructions will be displayed at the start of each session. Allow the subject about one hour to complete the task and as the task is self-paced, encourage the subjects to take breaks if needed.After all the sessions are completed, combine the raw data files and reformat them into a single JSON file for further processing by running preprocess. py in the terminal using the commands visible on the screen. When prompted, enter the requested input parameters including the path to the subject data directory, subject IDs to pre-process the data and the experiment name used to name the output file, then press Enter.This will create a JSON file in the output directory that combines responses across repeats for each trial. To determine the pair-wise choice probabilities from rank order judgments, go to similarities, then analysis and run describe_data. py in the command line.When prompted, enter the path to subject data and the list of subjects to run the analysis. This will create three kinds of plots. Generate low-dimensional Euclidean models of the perceptual spaces using the choice probabilities by running model_fitting.py using the command line displayed on the screen. When prompted, provide the inputs for the directory to the subject-data/preprocessed, the number of stimuli, which will be 37 by default, the number of iterations, the output directory, and the amount of Gaussian noise, which will be 0.18 by default. Visualize the log likelihood of the obtained models and assess their fit by running similarities, analysis, model_fitting_figure.py.When prompted, input the needed path to the CSV files containing log likelihoods. Visualize the perceptual spaces for each subject and generate scatter plots showing the points from the 5D model projected onto the first two principle components by running the displayed commands. When prompted, enter the input parameters and the path to the NPY file containing the 5D points.After the script is executed, exit the virtual environment. In the file generated for the word experiment, the first row corresponds to a trial in which eight stimuli appear around the reference stimulus monkey. The rank judgments were decomposed into pair-wise choices.The distribution of choice probabilities was highly consistent across the subjects. The data clustering near the diagonal in each panel indicates a great deal of consistency in the choice probabilities between the subjects and for the judgements that are not at the extremes. The dominant diagonal indicates that the choice probabilities in the two contexts, including the intermediate choice probabilities between zero and one, are close to identical for each subject.The log likelihoods are shown relative to the log likelihood of the best model, that is a model that assigns the observed choice probability to each comparison without constraining these probabilities by any geometric consideration. Principle component analysis was performed on the points from the 5D model of the perceptual space where the animals perceived as similar were denoted by points near each other. It is crucial to decide on all parameters when designing an experiment and to set them in the config files before beginning step two.Keeping careful track of each subject's data is also very important. The method provides a large number of similarity judgment, so a number of analyses could be applied, such as cluster analysis, a focus on context, or modeling with different geometric spaces.

Research

JoVE Journal

Neuroscience

Brain Imaging Investigation of the Impairing Effect of Emotion on Cognition

Emotions can impact cognition by exerting both enhancing (e.g., better memory for emotional events) and impairing (e.g., increased emotional ...

Organotypic Cerebellar Cultures: Apoptotic Challenges and Detection

Organotypic cultures of neuronal tissue were first introduced by Hogue in 1947 1,2 and have constituted a major breakthrough in the field of neuroscience. ...

Reproducible Mouse Sciatic Nerve Crush and Subsequent Assessment of Regeneration by Whole Mount Muscle Analysis

Regeneration in the peripheral nervous system (PNS) is widely studied both for its relevance to human disease and to understand the robust regenerative ...

Co-analysis of Brain Structure and Function using fMRI and Diffusion-weighted Imaging

The study of complex computational systems is facilitated by network maps, such as circuit diagrams. Such mapping is particularly informative when ...

Visualization and Genetic Manipulation of Dendrites and Spines in the Mouse Cerebral Cortex and Hippocampus using In utero Electroporation

Visualization and Genetic Manipulation of Dendrites and Spines in the Mouse Cerebral Cortex and Hippocampus using In utero Electroporation

In utero electroporation (IUE) has become a powerful technique to study the development of different regions of the embryonic nervous system 1-5. To date ...

A Visual Description of the Dissection of the Cerebral Surface Vasculature and Associated Meninges and the Choroid Plexus from Rat Brain

This video presentation was created to show a method of harvesting the two most important highly vascular structures, not residing within the brain ...

A Novel Light Damage Paradigm for Use in Retinal Regeneration Studies in Adult Zebrafish

Light-induced retinal degeneration (LIRD) is commonly used in both rodents and zebrafish to damage rod and cone photoreceptors. In adult zebrafish, ...

High-resolution Quantification of Odor-guided Behavior in Drosophila melanogaster Using the Flywalk Paradigm

High-resolution Quantification of Odor-guided Behavior in Drosophila melanogaster Using the Flywalk Paradigm

In their natural environment, insects such as the vinegar fly Drosophila melanogaster are bombarded with a huge amount of chemically distinct odorants. To ...

Chronic Implantation of Whole-cortical Electrocorticographic Array in the Common Marmoset

Electrocorticography (ECoG) allows the monitoring of electrical field potentials from the cerebral cortex with high spatiotemporal resolution. Recent ...

3D Kinematic Analysis for the Functional Evaluation in the Rat Model of Sciatic Nerve Crush Injury

Compared to the Sciatic Functional Index (SFI), kinematic analysis is a more reliable and sensitive method for performing functional evaluations of ...