Development of an Audio-based Virtual Gaming Environment to Assist with Navigation Skills in the Blind

Summary

Audio-based Environment Simulator (AbES) is virtual environment software designed to improve real world navigation skills in the blind.

Abstract

Audio-based Environment Simulator (AbES) is virtual environment software designed to improve real world navigation skills in the blind. Using only audio based cues and set within the context of a video game metaphor, users gather relevant spatial information regarding a building's layout. This allows the user to develop an accurate spatial cognitive map of a large-scale three-dimensional space that can be manipulated for the purposes of a real indoor navigation task. After game play, participants are then assessed on their ability to navigate within the target physical building represented in the game. Preliminary results suggest that early blind users were able to acquire relevant information regarding the spatial layout of a previously unfamiliar building as indexed by their performance on a series of navigation tasks. These tasks included path finding through the virtual and physical building, as well as a series of drop off tasks. We find that the immersive and highly interactive nature of the AbES software appears to greatly engage the blind user to actively explore the virtual environment. Applications of this approach may extend to larger populations of visually impaired individuals.

Introduction

Finding one's way in an unfamiliar environment presents as a significant challenge for the blind. Navigating successfully requires an understanding of the spatial relationships that exist between one's self and objects in the environment^1,2. The mental representation that describes surrounding space is referred to as a spatial cognitive map³. Blind individuals can gather relevant spatial information regarding their surrounding environment through other sensory channels (such as hearing) allowing for the generation of an accurate spatial cognitive map for the purposes of real world navigation tasks^4,5.

Considerable interest has arisen regarding the educative potential of virtual environments and action video games as a means to learn and master skills^6-9. Indeed, many strategies and approaches have been developed for the blind for this purpose (see ^4,10-12). We have developed Audio-based Environment Simulator (AbES); a user-centered audio-based virtual environment that allows for simulated navigation and exploration of an existing physical building. Drawing from original architectural floor plans, a virtual rendering of a modern two-story building (located at the Carroll Center for the Blind; Newton, MA) was generated with the AbES software (Figures 1A and B). AbES incorporates an action game metaphor with a premise designed to promote full exploration of the building space. Using simple key strokes and spatialized sound cues, users navigate and explore the entire building to collect a maximum number of jewels hidden in various rooms. Users must avoid roving monsters that can take them away and hide them elsewhere in the building (Figure 1C).

We demonstrate that interacting with AbES allows a blind user to generate an accurate spatial cognitive map of a target building based on auditory information acquired within the context of an action game metaphor. This is confirmed by a series of post-training behavioral performance tests designed to assess the transfer of acquired spatial information from a virtual environment to a real-world and large scale indoor navigation task (see Figure 2 for overall study design). Our results show that blind users are able to successfully navigate throughout a building for which they were previously unfamiliar, despite the fact that at no time were they informed of the overall purpose of the study, nor were they instructed to recall the spatial layout of the building while playing the game.

Protocol

1. Participant Demographics

This is an on-going study that recruits blind male and female participants aged between 18-45 years. All participants are legally blind of early onset (documented prior to the age of 3) and of varying ocular etiologies. None of the study participants were previously familiar with the spatial layout of the target physical building.

2. Preparation and Familiarization with AbES

1. Provide the participant with a blindfold and headphones to be worn throughout the training and assessment process. Ensure that the blindfold is comfortably placed over the eyes and the headphones are properly oriented and positioned over the ears (i.e. left speaker over left ear).
2. Train participant how to use assigned keys and the information represented by the audio cues in AbES. Using specific key strokes (Figure 3), a user navigates through and explores the building virtually (moving forward, right or left). Each virtual step approximates one step in the real physical building.
3. Familiarize with rules and premise of the game.
4. Familiarize with audio cues specific to game play (e.g. sound of locating jewels and sound of monsters nearby). As the user navigates through the building, auditory-based and contextual spatial information is acquired sequentially and is dynamically updated. Spatial and situational information is based on iconic and spatialized sound cues provided after each step taken. Orientation is based on cardinal compass headings (e.g. "north" or "east") and text through speech (TTS) is used to provide further information regarding a user's current location, orientation and heading (e.g. "you are in the corridor on the first floor, facing west") as well as the identity of objects and obstacles in their path (e.g. "this is a door"). Distance cues are provided based on modulating sound intensity. The spatial localization of the sounds is updated to match the user's egocentric heading. Essentially, the software is designed to play an appropriate audio file as a function of the location and egocentric heading of the user and keeps track of the user's position as they move through the environment. For example, if a door is located on the person's right side, the knocking sound is heard in the user's right ear (i.e. the software plays an audio file of a knocking sound in the right channel). If the person now turns around 180 degrees so that the same door is now located on their left side, the same knocking sound is now heard in the left channel (i.e. the software plays an audio file of a knocking sound in the left channel). Finally, if the user is facing the door, the same knocking sound is heard in both ears equally. By keeping track of the user's egocentric heading, the software can play the appropriate spatial localized sounds that identify the presence and location of objects and keep track of these changes as the user moves through the virtual environment. See Figure 4.

3. Training and Game Play with AbES (3 Sessions Each Lasting 30 min for a Total of 1.5 hr)

1. Allow for free game play and note any difficulties and challenges (i.e. use of key strokes, audio cues, areas of difficult navigation). Positive reinforcement and clarifications are provided at the end of each training session.
2. Record game performance (e.g. number, time and location where a participant finds a jewel).

4. Assess Virtual Navigation Task Performance

1. Explain to participant the details of the testing and provide instructions on how to complete the virtual navigation tasks. The participant will complete 10 predetermined navigation tasks presented sequentially using the AbES software (i.e. once the participant successfully completes the first task, the computer will automatically re-locate them to the starting point of the following task).
2. Inform the participant that they will have a maximum of 6 min to complete each navigation task.
3. 10 virtual navigation paths of comparable difficulty (i.e. distance traveled and number of turns) are chosen based on predetermined pairings of 10 start and stop locations (i.e. rooms). Specifically, the range of steps needed to navigate the target route ranged between 25-35 steps (in the virtual environment) and incorporated between 3-4 turns of 90 degrees.
4. Load the 10 navigation pairs into AbES for automated presentation and data capture of performance.
5. Outcome measures are automatically recorded using AbES' internal software. Outcome measures include: successful completion of the navigation task and time taken to reach target. See Figure 5A.
6. Instructions describing the start location and the target destination are provided automatically by the AbES software at the start of each task. Timing begins immediately once the subject takes their first virtual step from the starting location and ends once arriving at the target location (unless time takes longer than 6 min, for which the run is scored as incomplete and the next path is presented). Captured data is automatically sent to a text file and opened subsequently in database/statistical software for further analysis.

5. Assess Physical Navigation Task Performance

1. Explain to participant the details of the testing and provide instructions on how to complete the physical navigation tasks. The participant will complete 10 predetermined navigation tasks (presented in scrambled order from the previous virtual performance assessment) and under the supervision of an experienced investigator.
2. Inform the participant they will have a maximum of 6 min to complete each navigation task. For the purposes of the physical navigation task, the participant is allowed to use their white cane for mobility support.
3. 10 physical navigation paths are chosen based on predetermined pairings of 10 start and stop locations (i.e. rooms) of comparable difficulty (i.e. distance traveled and number of turns).
4. Investigator prepares stopwatch and clipboard with list of navigation tasks for manual scoring of performance.
5. Outcome measures are manually recorded by the investigator. Outcome measures include: successful completion of the navigation task and time taken to reach target.
6. "Square-off" the participant (i.e. position the participant with the door of the starting location behind them). Instructions describing the start location and the target destination are provided by the investigator at the start of each task. Timing begins immediately once the subject takes their first physical step from the starting location and ends when the participant verbally reports arriving at the destination (unless time takes longer than 6 min, for which the run is scored as an incomplete and the next path is presented). Captured data is recorded manually and subsequently transferred to database/statistical software for further analysis. See Figure 5B.

6. Assess Physical Drop off Task Performance

1. Explain to participant the details of the testing and provide instructions on how to complete the physical drop off navigation tasks. The participant will complete 5 navigation tasks with the goals of exiting the building using the shortest route possible and under the supervision of an experienced investigator.
2. Inform the participant that they will have a maximum of 6 min to complete each navigation task. For the purposes of the physical drop off navigation task, the participant is allowed to use their white cane for mobility support.
3. 5 predetermined physical starting locations are used such that 3 exit paths of different lengths are possible.
4. Investigator prepares stopwatch and clipboard with list of navigation tasks for manual scoring of performance.
5. Outcome measures are manually recorded by the investigator. Outcome measures include: successful completion of the navigation task and time taken to reach target. Furthermore, paths are scored such that the shortest path taken is given maximum points (i.e. 3 for shortest path, 2 for the second, 1 for the longest, and 0 for not being able to complete the task). See Figure 5C.
6. "Square-off" the participant at the first starting location. Instructions describing the start location are provided by the investigator at the start of each task. Timing begins immediately once the subject takes their first physical step from the starting location and ends when the participant verbally reports arriving to an exit door of the building (unless time takes longer than 6 min, for which the run is scored as incomplete and the next start location is presented). Captured data is recorded manually and subsequently transferred to database/statistical software for further analysis.

Results

Results from three early blind participants (aged between 19 and 22 years) are shown (see Table 1 for participant characteristics). In summary, all three participants showed a high level of success on all three navigation tasks following game play with the AbES software. This was confirmed by the performance scores (group mean and individual) on all three behavioral tasks (see Figure 6). The percentage correct performance for the virtual (mean: 90%) followed by the physical (mean: 88.7%)...

Discussion

We describe an interactive audio-based virtual environment simulator designed to improve general spatial awareness and navigation skills in the blind. We demonstrate that interacting with AbES provides accurate cues that describe the spatial relationships between objects and the overall layout of the target environment. Blind users can generate accurate spatial cognitive maps based on this auditory information and by interacting with the immersive virtual environment. Furthermore, interacting with AbES within the context...

Materials

Name	Company	Catalog Number	Comments
Name of Equipment	Comments
Laptop computer	Laptop used exclusively for training participants and collecting data
Stereo Head phones (fully enclosed circumaural design)	Worn by all participants during training
Blindfold	Worn by all participants during training and testing