Reliability in Psychology Experiments

Overview

Source: Laboratories of Gary Lewandowski, Dave Strohmetz, and Natalie Ciarocco—Monmouth University

In order to study something scientifically, a researcher needs to determine a way to quantify it. However, psychological constructs can be challenging to measure and quantify. This video examines reliability in the context of content analysis.

A recent study in the journal Pediatrics reported that 4-year-olds who watched a fast-paced cartoon had worse performance on cognitive tasks, such as following rules in a game, listening to direction from an adult, and delaying gratification, compared to other children who watched a slower paced cartoon.¹ In addition to the pace of the cartoon, the content of the cartoon may also have deleterious effects on its young viewers.

This video uses a simple two-group design, to exemplify the issue of reliability, in examining the question of whether the cartoon SpongeBob SquarePants has more inappropriate content than does the cartoon Caillou.

Procedure

1. Define key variables.

Create an operational definition (i.e., a clear description of exactly what a researcher means by a concept) of inappropriate content.
Consult definitions created by the organization TV Parental Guidelines and approved by the Federal Communications Commission.
Inappropriate content is any crude or rude behavior (e.g., toilet humor), depictions of verbal or physical aggression (e.g., name calling, hitting, etc.), bad language (
Log in or to access full content. Learn more about your institution’s access to JoVE content here

Results

The results indicate that the raters had a high level of agreement or consistency in their ratings within each cartoon episode, which indicates high inter-rater reliability (Figure 1). There is also reliability or consistency in SpongeBob SquarePants episodes having more inappropriate content than Caillou. The results also revealed individual biases amongst raters. For example, Rater 3 reported more inappropriate content in SpongeBob than the other 2 raters, and Rater 1 reported less in Caillou than othe

Application and Summary

Researchers have increasingly turned their attention toward analyzing television’s content, especially as it relates to children. As discussed prior to this current experiment, a recent study in the journal Pediatrics correlated the fast pace of the SpongeBob SquarePants cartoon to relatively poor cognitive abilities in the children who watch it.

Since the results of our experiment appear reliable, future research could examine whether the relative amount of inappropriate conten

References

Lillard, A. S., & Peterson, J. The Immediate Impact of Different Types of Television on Young Children's Executive Function. Pediatrics. 128(4):644-9. doi: 10.1542/peds.2010-1919 (2011).