Name: a psychophysics paradigm for the collection and analysis of similarity judgments
Uploaded: 2022-03-01T13:45:00
Description: Watch this Scientific Journal Video about A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments at JoVE.com

The work is supported by funding from the National Institutes of Health (NIH), grant EY07977. The authors would also like to thank Usman Ayyaz for his assistance in testing the software, and Muhammad Naeem Ayyaz for his comments on the manuscript.

Prior to beginning the experiments, all subjects provide informed consent in accordance with institutional guidelines and the Declaration of Helsinki. In the case of this study, the protocol was approved by the institutional review board of Weill Cornell Medical College.
1. Installation and set-up
<ol>
	<li>Download the code from the GitHub repository, similarities (https://github.com/jvlab/similarities). In the command line, run:&#160;git clone https://github.com/jvlab/...

<ol>
	<li>Edelman, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Representation+is+representation+of+similarities.">Representation is representation of similarities.</a> TheBehavioral and Brain Sciences. 21 (4), 449-498 (1998).</li><li>Hahn, U., Chater, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Concepts+and+similarity.">Concepts and similarity.</a> Knowledge, Concepts and Categories. , 43-84 (1997).</li><li>Kriegeskorte, N., Kievit, R. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Representational+geometry:+integrating+cognition,+computation,+and+the+brain.">Representational geometry: integrating cognition, computation, and the brain.</a> Trends in Cognitive Sciences. 17 (8), 401-412 (2013).</li><li>Hebart, M. N., Zheng, C. Y., Pereira, F., Baker, C. I. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Revealing+the+multidimensional+mental+representations+of+natural+objects+underlying+human+similarity+judgements.">Revealing the multidimensional mental representations of natural objects underlying human similarity judgements.</a> Nature Human Behaviour. 4 (11), 1173-1185 (2020).</li><li>Deng, W. S., Sloutsky, V. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+development+of+categorization:+Effects+of+classification+and+inference+training+on+category+representation.">The development of categorization: Effects of classification and inference training on category representation.</a> Developmental Psychology. 51 (3), 392-405 (2015).</li><li>Shepard, R. N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Stimulus+and+response+generalization:+tests+of+a+model+relating+generalization+to+distance+in+psychological+space.">Stimulus and response generalization: tests of a model relating generalization to distance in psychological space.</a> Journal of Experimental Psychology. 55 (6), 509-523 (1958).</li><li>Coombs, C. H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+method+for+the+study+of+interstimulus+similarity.">A method for the study of interstimulus similarity.</a> Psychometrika. 19 (3), 183-194 (1954).</li><li>Ga&#776;rdenfors, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Conceptual Spaces: The Geometry of Thought. , (2000).</li><li>Zaidi, Q., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Perceptual+spaces:+mathematical+structures+to+neural+mechanisms.">Perceptual spaces: mathematical structures to neural mechanisms.</a> The Journal of Neuroscience The Official Journal of the Society for Neuroscience. 33 (45), 17597-17602 (2013).</li><li>Krishnaiah, P. R., Kanal, L. N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Handbook of Statistics 2. , (1982).</li><li>Tsogo, L., Masson, M. H., Bardot, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Multidimensional+Scaling+Methods+for+Many-Object+Sets:+A+Review.">Multidimensional Scaling Methods for Many-Object Sets: A Review.</a> Multivariate Behavioral Research. 35 (3), 307-319 (2000).</li><li>Kriegeskorte, N., Mur, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Inverse+MDS:+Inferring+dissimilarity+structure+from+multiple+item+arrangements.">Inverse MDS: Inferring dissimilarity structure from multiple item arrangements.</a> Frontiers in Psychology. 3, 245 (2012).</li><li>Rao, V. R., Katz, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Alternative+Multidimensional+Scaling+Methods+for+Large+Stimulus+Sets.">Alternative Multidimensional Scaling Methods for Large Stimulus Sets.</a> Journal of Marketing Research. 8 (4), 488-494 (1971).</li><li>Hoffman, J. I. E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Hypergeometric+Distribution.">Hypergeometric Distribution.</a> Biostatistics for Medical and Biomedical Practitioners. , 179-182 (2015).</li><li>Victor, J. D., Rizvi, S. M., Conte, M. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Two+representations+of+a+high-dimensional+perceptual+space.">Two representations of a high-dimensional perceptual space.</a> Vision Research. 137, 1-23 (2017).</li><li>Knoblauch, K., Maloney, L. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Estimating+classification+images+with+generalized+linear+and+additive+models.">Estimating classification images with generalized linear and additive models.</a> Journal of Vision. 8 (16), 1-19 (2008).</li><li>Maloney, L. T., Yang, J. N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Maximum+likelihood+difference+scaling.">Maximum likelihood difference scaling.</a> Journal of Vision. 3 (8), 573-585 (2003).</li><li>Logvinenko, A. D., Maloney, L. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+proximity+structure+of+achromatic+surface+colors+and+the+impossibility+of+asymmetric+lightness+matching.">The proximity structure of achromatic surface colors and the impossibility of asymmetric lightness matching.</a> Perception &amp; Psychophysics. 68 (1), 76-83 (2006).</li><li>Zhou, Y., Smith, B. H., Sharpee, T. O. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Hyperbolic+geometry+of+the+olfactory+space.">Hyperbolic geometry of the olfactory space.</a> Science Advances. 4 (8), (2018).</li><li>Goldstone, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=An+efficient+method+for+obtaining+similarity+data.">An efficient method for obtaining similarity data.</a> Behavior Research Methods, Instruments, &amp; Computers. 26 (4), 381-386 (1994).</li><li>Townsend, J. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Theoretical+analysis+of+an+alphabetic+confusion+matrix.">Theoretical analysis of an alphabetic confusion matrix.</a> Perception &amp; Psychophysics. 9, 40-50 (1971).</li></ol>

The protocol outlined here is effective for obtaining and analyzing similarity judgments for stimuli that can be presented visually. The experimental paradigm, the analysis, and possible extensions are discussed first, and later the advantages and disadvantages of the method.
Experimental paradigm: The proposed method is demonstrated using a domain of 37 animal names, and a sample dataset of perceptual judgments is provided so that one can follow the analysis in step 5 and rep...

Humans mentally process and represent incoming sensory information to perform a wide range of tasks, such as object recognition, navigation, making inferences about the environment, and many others. Similarity judgments are commonly used to probe these mental representations1. Understanding the structure of mental representations can provide insight into the organization of conceptual knowledge2. It is also possible to gain insight into neural computations, by relating similarity judgments to brain activation patterns3. Additionally, similarity judgments reveal features that are salient in perception4. Studying how mental representations change during development can shed light on how they are learned5. Thus, similarity judgments provide valuable insight into information processing in the brain.
A common&#160;model of mental representations using similarities is a geometric space model6,7,8. Applied to sensory domains, this kind of model is often referred to as a perceptual space9. Points in the space represent stimuli and distances between points correspond to the perceived dissimilarity between them. From similarity judgments, one can obtain quantitative estimates of dissimilarities. These pairwise dissimilarities (or perceptual distances) can then be used to model the perceptual space via multidimensional scaling10.
There are many methods for collecting similarity judgments, each with its advantages and disadvantages. The most straightforward way of obtaining quantitative measures of dissimilarity is to ask subjects to rate on a scale the degree of dissimilarity between each pair of stimuli. While this is relatively quick, estimates tend to be unstable across long sessions as subjects cannot go back to previous judgments, and context effects, if present, cannot be detected. (Here, a context effect is defined as a change in the judged similarity between two stimuli, based on the presence of other stimuli that are not being compared.) Alternatively, subjects can be asked to compare all pairs of stimuli to all other pairs of stimuli. While this would yield a more reliable rank ordering of dissimilarities, the number of comparisons required scales with the fourth power of the number of stimuli, making it feasible for only small stimulus sets. Quicker alternatives, like sorting into a predefined number of clusters11&#160;or free sorting have their own limitations. Free sorting (into any number of piles) is intuitive, but it forces the subject to categorize the stimuli, even if the stimuli do not easily lend themselves to categorization. The more recent multi-arrangement method, inverse MDS, circumvents many of these limitations and is very efficient12. However, this method requires subjects to project their mental representations onto a 2D Euclidean plane and to consider similarities in a specific geometric manner, making the assumption that similarity structure can be recovered from Euclidean distances on a plane. Thus, there remains a need for an efficient method to collect large amounts of similarity judgments, without making assumptions about the geometry underlying the judgments.
Described here is a method that is both reasonably efficient and also avoids the above potential pitfalls. By asking subjects to rank stimuli in order of similarity to a central reference in each trial13, relative similarity can be probed directly, without assuming anything about the geometric structure of the subjects&#39; responses. The paradigm repeats a subset of comparisons with both identical and different contexts, allowing for direct assessment of context effects as well as the acquisition of graded responses in terms of choice probabilities. The analysis procedure decomposes these rank judgments into multiple pairwise comparisons and uses them to build and search for Euclidean models of perceptual spaces that explain the judgments. The method is suitable for describing in detail the representation of stimulus sets of moderate sizes (e.g., 19 to 49).
To demonstrate the feasibility of the approach, an experiment was conducted, using a set of 37 animals as stimuli. Data was collected over the course of 10 one-hour sessions and then analyzed separately for each subject. Analysis revealed consistency across subjects and negligible context effects. It also assessed consistency of perceived dissimilarities between stimuli with Euclidean models of their perceptual spaces. The paradigm and analysis procedures outlined in this paper are flexible and are expected to be of use to researchers interested in characterizing the geometric properties of a range of perceptual spaces.

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>Computer Workstation</td>
			<td>N/A</td>
			<td>N/A</td>
			<td>OS: Windows/ MacOS 10 or higher/ Linux; 3.1 GHz Dual-Core Intel Core i5 or similar; 8GB or more memory; User permissions for writing and executing files</td>
		</tr>
		<tr>
			<td>conda</td>
			<td></td>
			<td>Version 4.11</td>
			<td>OS: Windows/ MacOS 10 or higher/ Linux</td>
		</tr>
		<tr>
			<td>Microsoft Excel</td>
			<td>Microsoft</td>
			<td>Any</td>
			<td>To open and shuffle rows and columns in trial conditions files.</td>
		</tr>
		<tr>
			<td>PsychoPy</td>
			<td>N/A</td>
			<td>Version 2021.2</td>
			<td>Framework for running psychophysical studies</td>
		</tr>
		<tr>
			<td>Python 3</td>
			<td>Python Software Foundation</td>
			<td>Python Version 3.8</td>
			<td>Python3 and associated built-in libraries</td>
		</tr>
		<tr>
			<td>Required Python Libraries</td>
			<td>N/A</td>
			<td>numpy version: 1.17.2 or higher; matplotlib version 3.4.3 or higher; scipy version 1.3.1 or higher; pandas version 0.25.3 or higher; seaborn version 0.9.0 or higher; scikit_learn version 0.23.1 or higher; yaml version 6.0 or higher&#160;</td>
			<td>numpy, scipy and scikit_learn are computing modules with in-built functions for optimization and vector operations. matplotlib and seaborn are plotting libraries. pandas is used to reading in and edit data from csv files.</td>
		</tr>
	</tbody>
</table>

a psychophysics paradigm for the collection and analysis of similarity judgments

Similarity judgments are commonly used to study mental representations and their neural correlates. This approach has been used to characterize perceptual spaces in many domains: colors, objects, images, words, and sounds. Ideally, one might want to compare estimates of perceived similarity between all pairs of stimuli, but this is often impractical. For example, if one asks a subject to compare the similarity of two items with the similarity of two other items, the number of comparisons grows with the fourth power of the stimulus set size. An alternative strategy is to ask a subject to rate similarities of isolated pairs, e.g., on a Likert scale. This is much more efficient (the number of ratings grows quadratically with set size rather than quartically), but these ratings tend to be unstable and have limited resolution, and the approach also assumes that there are no context effects.
Here, a novel ranking paradigm for efficient collection of similarity judgments is presented, along with an analysis pipeline (software provided) that tests whether Euclidean distance models account for the data. Typical trials consist of eight stimuli around a central reference stimulus: the subject ranks stimuli in order of their similarity to the reference. By judicious selection of combinations of stimuli used in each trial, the approach has internal controls for consistency and context effects. The approach was validated for stimuli drawn from Euclidean spaces of up to five dimensions. 
The approach is illustrated with an experiment measuring similarities among 37 words. Each trial yields the results of 28 pairwise comparisons of the form, "Was A more similar to the reference than B was to the reference?" While directly comparing all pairs of pairs of stimuli would have required 221445 trials, this design enables reconstruction of the perceptual space from 5994 such comparisons obtained from 222 trials.

Figure 1A shows part of a conditions file generated by the script in step 3.3, for the word experiment. Each row corresponds to a trial. The stimulus in the ref column appears in the center of the display. The column names stim1 to stim8 correspond to eight positions along a circle, running counterclockwise, starting from the position to the right of the central reference. A sample trial from the word experiment is shown in Figure 1B.
<p cla...

Watch this Scientific Journal Video about A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments at JoVE.com

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments

The protocol presents an experimental psychophysics paradigm to obtain large quantities of similarity judgments, and an accompanying analysis workflow. The paradigm probes context effects and enables modeling of similarity data in terms of Euclidean spaces of at least five dimensions.

Similarity judgments are commonly used to study mental representations and their neural correlates. This approach has been used to characterize perceptual ...

Research

JoVE Journal

Neuroscience

Organotypic Cerebellar Cultures: Apoptotic Challenges and Detection

Organotypic cultures of neuronal tissue were first introduced by Hogue in 1947 1,2 and have constituted a major breakthrough in the field of neuroscience. ...

Reproducible Mouse Sciatic Nerve Crush and Subsequent Assessment of Regeneration by Whole Mount Muscle Analysis

Regeneration in the peripheral nervous system (PNS) is widely studied both for its relevance to human disease and to understand the robust regenerative ...

Co-analysis of Brain Structure and Function using fMRI and Diffusion-weighted Imaging

The study of complex computational systems is facilitated by network maps, such as circuit diagrams. Such mapping is particularly informative when ...

Visualization and Genetic Manipulation of Dendrites and Spines in the Mouse Cerebral Cortex and Hippocampus using In utero Electroporation

Visualization and Genetic Manipulation of Dendrites and Spines in the Mouse Cerebral Cortex and Hippocampus using In utero Electroporation

In utero electroporation (IUE) has become a powerful technique to study the development of different regions of the embryonic nervous system 1-5. To date ...

A Visual Description of the Dissection of the Cerebral Surface Vasculature and Associated Meninges and the Choroid Plexus from Rat Brain

This video presentation was created to show a method of harvesting the two most important highly vascular structures, not residing within the brain ...

A Novel Light Damage Paradigm for Use in Retinal Regeneration Studies in Adult Zebrafish

Light-induced retinal degeneration (LIRD) is commonly used in both rodents and zebrafish to damage rod and cone photoreceptors. In adult zebrafish, ...

High-resolution Quantification of Odor-guided Behavior in Drosophila melanogaster Using the Flywalk Paradigm

High-resolution Quantification of Odor-guided Behavior in Drosophila melanogaster Using the Flywalk Paradigm

In their natural environment, insects such as the vinegar fly Drosophila melanogaster are bombarded with a huge amount of chemically distinct odorants. To ...

Chronic Implantation of Whole-cortical Electrocorticographic Array in the Common Marmoset

Electrocorticography (ECoG) allows the monitoring of electrical field potentials from the cerebral cortex with high spatiotemporal resolution. Recent ...

3D Kinematic Analysis for the Functional Evaluation in the Rat Model of Sciatic Nerve Crush Injury

Compared to the Sciatic Functional Index (SFI), kinematic analysis is a more reliable and sensitive method for performing functional evaluations of ...

Collection of Frozen Rodent Brain Regions for Downstream Analyses

As our understanding of neurobiology has progressed, molecular analyses are often performed on small brain areas such as the medial prefrontal cortex ...