Authors
Contact Us

Sign In

In This Article

  • Summary
  • Abstract
  • Introduction
  • Protocol
  • Representative Results
  • Discussion
  • Acknowledgements
  • Materials
  • References
  • Reprints and Permissions

Summary

Virtual reality (VR) is a powerful yet underutilized approach to advance the diagnosis and treatment of neurological disease. The Cleveland Clinic Virtual Reality Shopping platform combines state-of-the-art VR content with an omnidirectional treadmill to quantify instrumental activities of daily living-a proposed prodromal marker of neurological disease.

Abstract

A decline in the performance of instrumental activities of daily living (IADLs) has been proposed as a prodromal marker of neurological disease. Existing clinical and performance-based IADL assessments are not feasible for integration into clinical medicine. Virtual reality (VR) is a powerful yet underutilized tool that could advance the diagnosis and treatment of neurological disease. An impediment to the adoption and scaling of VR in clinical neurology is VR-related sickness resulting from sensory inconsistencies between the visual and vestibular systems (i.e., locomotion problem).

The Cleveland Clinic Virtual Reality Shopping (CC-VRS) platform attempts to solve the locomotion problem by coupling an omnidirectional treadmill with high-resolution VR content, enabling the user to physically navigate a virtual grocery store to simulate shopping. The CC-VRS consists of Basic and Complex shopping experiences; both require walking 150 m and retrieving five items. The Complex experience has additional scenarios that increase the cognitive and motor demands of the task to better represent the continuum of activities associated with real-world shopping. The CC-VRS platform provides objective and quantitative biomechanical and cognitive outcomes related to the user's IADL performance. Initial data indicate that the CC-VRS results in minimal VR-sickness and is feasible and tolerable for older adults and patients with Parkinson's disease (PD). The considerations underlying the development, design, and hardware and software technology are reviewed, and initial models of integration into primary care and neurology are provided.

Introduction

In 2008, the National Academy of Engineering identified 14 Grand Challenges for Engineering in the 21st Century1. One of those was the integration of virtual reality (VR) into medicine. Progress has been made in the use of VR for training for medical students2,3, surgical planning3, reducing anxiety associated with medical interactions4, assisting in the management of acute5 and cancer-related pain6, and augmenting motor recovery following stroke7. Despite these promising applications, the utility of VR in medicine has not been fully realized, particularly in the realm of evaluating and treating neurological disease. While advances in VR technology have minimized barriers such as cost, headset comfort, and intuitive usability features, VR sickness continues to impede the integration of VR into medicine8.

Virtual reality sickness refers to feelings akin to motion sickness (e.g., nausea, vomiting, vertigo)9,10,11 that arise during VR experiences. Although no single theory is agreed upon in explaining VR sickness, the Sensory Conflict Theory is a leading explanation12. Briefly, the Sensory Conflict Theory suggests that VR sickness arises from sensory disparities; visual flow information indicates the body's forward movement through space while the vestibular system indicates the body is stationary13. This discrepancy in sensory information results in poor balance, spatial disorientation, and uncontrollable postural movements that are precursors to VR sickness. While the precise mechanism underlying VR sickness is debated, reducing the mismatch between sources of sensory information is likely to reduce VR sickness14 and facilitate VR adoption in a medical setting.

Locomotion coupled with VR has long been proposed as an approach to reducing sensory mismatch by both physically and visually immersing the user in the virtual environment15,16. Several studies in older adults with and without neurological disease have successfully paired immersive and non-immersive VR systems with traditional unidirectional treadmills17,18,19. These studies demonstrate that a VR and unidirectional treadmill intervention is typically well-tolerated18 and the intervention may reduce fall frequency17,19. These results provide a promising foundation for the successful integration of locomotion and VR. However, the external motor pacing of a unidirectional treadmill does not allow the user to change speeds or execute turns to interact with more complex realistic virtual environments.

Over the past two decades, advances in movement-tracking hardware and software have facilitated the development of more immersive and interactive virtual environments. A major advancement has been the development of the omnidirectional treadmill20. Briefly, an omnidirectional treadmill simultaneously utilizes linear and rotational movements to enable the user to ambulate in any direction at a self-selected pace. Generally utilized in the gaming industry, omnidirectional treadmills broaden opportunities to leverage VR environments in the clinical setting by both addressing the VR sickness problem and facilitating the creation of realistic environments that better challenge the physical capabilities of the user, such as turning or changing directions. In particular, virtual replications of full-scale, everyday environments can facilitate the evaluation of cognitive and motor functioning during the performance of instrumental activities of daily living (IADLs).

Instrumental activities of daily living (IADLs) are functional tasks (e.g., shopping, taking medication, food preparation) that are critical for maintaining independent living21. The ability to accomplish common IADLs has been proposed as a prodromal marker for neurological disease. Recent data from long-term, prospective studies indicate declines in IADLs likely precede a diagnosis of Parkinson's disease (PD) by 5-7 years22,23 and a diagnosis of Alzheimer's disease24,25. In contrast to basic activities of daily living (BADLs)26, IADLs typically require the simultaneous performance of two attention-demanding tasks (e.g., motor-cognitive, motor-motor, or cognitive-cognitive)27. The vast majority of daily household and community activities are performed under dual-task conditions28,29.

Although dual-task declines clearly impact IADL performance, traditional clinical motor evaluations30,31,32 and neuropsychological tests33,34 are insufficient to evaluate IADLs, as these assessments separate function into discrete components without consideration of their interdependence. Current methods of direct IADL assessment rely on bias-prone self-report questionnaires35 or lengthy and burdensome performance-based evaluations36. Neither approach provides objective, quantitative insights into an individual's level of IADL function in the community setting.

Advances in VR technology, coupled with the engineering advances underlying omnidirectional treadmills, provide an opportunity to create an interactive and immersive environment. A virtual grocery store and shopping task were created to simultaneously assess motor, cognitive, cognitive-motor, and IADL performance. The Cleveland Clinic Virtual Reality Shopping (CC-VRS) platform was collaboratively developed by a team of biomedical engineers, software developers, physical therapists, occupational therapists, and neurologists.

A grocery shopping task was selected to quantify IADL performance based on recommendations from the American Occupational Therapy Association26. The Virtual Multiple Errands Task (VMET)37, Timed Instrumental ADL Scale38, and Penn Parkinson’s Daily Activities Questionnaire-15 (PDAQ-15)39 recognize shopping as an important indicator of motor and non-motor performance associated with neurological disease. Others have used an immersive VR headset to create a grocery store environment in an attempt to estimate IADL performance37,40,41. However, they have failed to evaluate a major component of grocery shopping: locomotion. Generally, current VR grocery store paradigms require the participant to use a hand-held controller to teleport or navigate an avatar throughout the grocery store. We aimed to integrate locomotion into the virtual shopping experience of the user. The CC-VRS development process began with a formal task analysis of a typical grocery store experience. As indicated in Figure 1, nine fundamental task components reflect a blend of elements that can be characterized as motor, cognitive, or cognitive-motor activities necessary for successful performance, as is characteristic of all IADLs.

figure-introduction-8230
Figure 1: Grocery shopping task analysis. A task analysis was performed to identify the sequence of actions and the nature of those actions for successful grocery shopping in the real world. Nine primary sequences were identified and were used to inform the development of the Basic and Complex shopping tasks. The sequences were classified as motor (blue), cognitive (yellow), and cognitive-motor (green); details regarding corresponding outcomes are provided in Table 1. Please click here to view a larger version of this figure.

The CC-VRS platform replicates a realistic, medium-sized grocery store via an immersive VR headset. While walking on an omnidirectional treadmill, the user follows a continuous, designated route through the store, locates items on a shopping list, and places the items in a virtual shopping cart. Providing a designated route standardizes the distance walked through the virtual store, reduces the number of navigational errors, and facilitates greater precision in dissociating potential changes in IADL performance from navigational errors or suboptimal search strategies employed by the user. The 150 m route requires multiple turns, which increases motor complexity42,43 and the probability of triggering freezing of gait in neurological patient populations, as freezes are more frequently observed during turning than straight line walking44,45. Both distance of the navigational path and number of items on the shopping list can be configured by the clinician to match the abilities of the user or goals of the assessment session.

Each user completes one Basic and one Complex shopping scenario. The Basic Scenario simply requires following the route and selecting items from the shopping list. In the Complex Scenario, the user is provided a list of different grocery items while following the identical route through the store, but additional cognitive and motor demands are introduced (delayed verbal recall, price comparison, and obstacle avoidance tasks described in the protocol section below). Ambient grocery store noise throughout both the Basic and Complex Scenarios completes the immersive experience. Summary and detail data on the user's performance—including correct and incorrect items gathered, number and frequency of shopping list activations, stop duration, and gait metrics—are automatically generated and available for review by the clinician.

The goal of the CC-VRS is to objectively quantify the performance of IADLs in older adults and individuals at risk for or diagnosed with neurological disease. The CC-VRS provides an immersive and realistic experience for the user, and it yields precise, biomechanically-based outcomes of cognitive and motor function that have the potential to serve as prodromal markers of neurological disease or objective measures of disease progression. The CC-VRS is currently being used in three related projects aimed at: (1) understanding the effects of healthy aging and neurological disease on IADL performance, (2) determining the feasibility of clinical integration into primary care and a movement disorder clinic, and (3) identifying the neural signature underlying freezing of gait in advanced PD patients with deep brain stimulation (DBS) systems. Collectively, these projects will utilize the CC-VRS platform and associated outcomes to better understand how aging and neurological disease impact aspects of IADL performance. This manuscript details the development, design, and hardware and software technology of the CC-VRS and its novel outcomes that can facilitate integration into health care.

Protocol

The outlined protocol follows the guidelines of the Cleveland Clinic human research ethics committee. All participants completed the informed consent process and provided written permission to publish photos taken during data collection.

1. Equipment setup and calibration (5 min)

  1. VR system
    1. Ensure the system includes all components outlined in the experimental setup diagrammed in Figure 2, including a VR headset, two hand controllers, one VR waist tracker, two VR foot trackers, base stations to monitor the position of the VR devices, and a high-end gaming desktop with a 2080ti graphic card to run the VR system and CC-VRS software (see the Table of Materials).
      1. Launch Steam VR on the desktop to coordinate the VR components and monitor the status of each VR device throughout data collection.
      2. Turn on each VR device and look for a green indicator light to verify active tracking by Steam VR.
      3. Calibrate the boundaries and orientation of the virtual space by selecting Room Set Up in the Steam VR menu and following on-screen prompts using the hand controllers.
  2. VR headset
    1. Place the headset in the UV hygienic cleaning system and run one sanitizing cycle between users.
  3. Omnidirectional treadmill
    1. Power on the omnidirectional treadmill using the green power button on the attached foot pedal. Launch the corresponding software on the desktop computer.
    2. To calibrate, use the Select User Tracker function in the application and identify the waist tracker as the appropriate tracking device. Next, center this tracker on the omnidirectional treadmill surface and use the Set Center Point function to calibrate the middle of the treadmill platform. Set the waist tracker on the ring and use the Set Ring Height function to calibrate the height of the handrail. 
      NOTE: The treadmill and corresponding software rely on the position of the VR waist tracker relative to the platform to operate appropriately in response to the user's movements. The user begins stationary, positioned in the center of the treadmill. As the user moves off-center, the system responds to the user's movements and speed by generating the appropriate treadmill motion that will recenter the user on the platform.
  4. CC-VRS application
    1. When all VR tracking devices and the omnidirectional treadmill are calibrated and engaged, launch the CC-VRS application from the desktop. Follow the on-screen menus to input the User ID and initiate the appropriate trial type.

figure-protocol-2946
Figure 2: CC-VRS platform overview. A depiction of the entire CC-VRS platform. The user wears a VR headset and navigates through a virtual grocery store by walking on the omnidirectional treadmill. A subtle green line is provided to the user via the VR headset as a navigational aid. The five items on the shopping list can be found along this provided 150 m path. A first-person view of the user is provided to the experimenter via the Control Computer and Monitor. The time necessary to set up the CC-VRS system is approximately 5 min. Abbreviations: VR = virtual reality; CC-VRS = Cleveland Clinic-Virtual Reality Shopping. Please click here to view a larger version of this figure.

2. Preparation of user (15 min)

  1. Tolerability survey (baseline)
    1. If collecting data about VR sickness, instruct the user to complete the Simulator Sickness Questionnaire before beginning the CC-VRS experience.
  2. Harness
    1. Fit the user in a full-body harness that secures around the legs and chest. Clip the harness into a ceiling-mounted cable above the center of the omnidirectional treadmill to prevent falls and increase the level of comfort for the user without impeding natural gait.
  3. VR trackers
    1. Affix the left and right foot trackers to the user's feet by using zip ties around the shoelaces. Screw the waist tracker onto the specially designed waist belt and adjust the belt until the tracker sits in the middle of the user's lumbar region. Place the left and right controllers in the user's hands and tighten the straps until secure and comfortable.
  4. Omnidirectional treadmill familiarization
    1. Before donning the headset, allow the user time to walk and turn on the omnidirectional treadmill. Explain the importance of the waist tracker position relative to the center of the treadmill platform, and encourage the user to get comfortable walking toward the outer edges of the treadmill boundary while holding the handrail for support. Disengage the treadmill via the application to continue with user preparation.
  5. Headset
    1. With the user standing on the stationary omnidirectional treadmill, place the headset on the user's head and assist with adjustments (top weight-bearing strap, rear stability knob, and interpupillary distance slider for clarity) until the fit is comfortable and the display is clear. Ensure the headset-mounted speakers are positioned over the ears and set to an appropriate volume level.
  6. Instruct the user to stand near the center of the omnidirectional treadmill platform and click Start on the application to re-engage the treadmill.
  7. Launch the CC-VRS application from the desktop if not launched previously.

3. Administration of CC-VRS (30 min)

  1. Throughout the CC-VRS experience, monitor the user's progress through the store via the desktop display and be prepared to stop the omnidirectional treadmill should the user experience any discomfort or instability.
  2. Enter User ID.
  3. Select Comprehensive Tutorial to load a small practice environment that introduces the user to the overall goal of the CC-VRS assessment, in addition to the navigational route, shopping list, and additional cognitive demands of the Complex Scenario.
    1. Ensure the user is comfortable with the following controller functions before proceeding with testing:
      1. Activate the shopping list by raising the left hand and holding the A or B button on the controller (Figure 3A).
      2. Close the shopping list by releasing the A or B button.
      3. Select items from the shelves using the controller triggers (Figure 3A).
      4. Place items into the grocery cart using the controller triggers.
    2. Ensure the user is comfortable with the following cognitive and motor demands of the Complex Scenario:
      1. Execute a delayed verbal recall of five words presented via an auditory announcement at the start of the Scenario, similar to the Delayed Recall component of the Montreal Cognitive Assessment test (MoCA)46.
      2. Perform a price comparison task for sale items (e.g., choosing the most cost-effective option between 8 oz of ketchup for $1.00 vs 16 oz for $1.50) (Figure 3B).
      3. Avoid obstacles in the store, including spills on the floor and narrowed aisleways caused by placement of other shoppers or carts along the path (Figure 3C).
    3. If necessary, repeat the tutorial (approximately 5 min total) until the user demonstrates proficiency with the above functions and understanding of the task.
  4. Select Basic Scenario. Choose path length and the number of list items.
    1. Instruct the user to begin walking as soon as the store is visible on the headset display. Encourage the user to complete the task as efficiently as possible, moving quickly while minimizing errors.
    2. When the user has completed the task by reaching the store checkout, review the summary metrics displayed on the desktop screen and exit the virtual environment.
  5. Select Complex Scenario. Choose the path length and the number of list items.
    1. Provide similar instructions to the user as in the Basic Scenario. Remind the user of the additional cognitive demands in the Complex Scenario.
    2. When the user has completed the task by reaching the store checkout, review the summary metrics displayed on the desktop screen (Figure 3D) and exit the virtual environment.
  6. Tolerability survey
    1. If collecting data about VR sickness, instruct the user to complete the Simulator Sickness Questionnaire immediately upon completion of the CC-VRS experience and again up to 30 min later.
  7. Usability survey
    1. If collecting data about usability of the platform, instruct the user to complete the System Usability Scale immediately upon completion of the CC-VRS.

figure-protocol-9910
Figure 3: CC-VRS environment. (A) A first-person view of a CC-VRS user actively viewing the shopping list with the left hand and selecting a corresponding item with the right hand. Users can intuitively interact with any item throughout the grocery store by using VR hand controllers. (B) An example of a sale price comparison task the user encounters in the Complex Scenario. For an item on the shopping list denoted as a SALE item, the user must compare the unit prices of two differently sized items and select the option that represents the better deal. (C) A first-person view of a narrowed aisle found in the Complex Scenario. In addition to the multiple turns along the navigational route, the tight passageways add motor complexity that increases the probability of triggering freezing of gait in neurological populations. (D) An example of the summary outcomes displayed to the experimenter upon completion of a Complex Scenario, including correct and incorrect items, total time to complete the scenario, and number of words successfully recalled. The specific metrics in this display are configurable by the experimenter. Abbreviations: VR = virtual reality; CC-VRS = Cleveland Clinic-Virtual Reality Shopping. Please click here to view a larger version of this figure.

4. Data files and results

  1. Review the summary file (.csv) automatically generated for each trial, which contains configurable metrics to characterize the overall performance of the CC-VRS.
  2. Review the detailed data file (.csv) that contains the position and rotation of the trackers, controllers, and headset throughout the duration of the task. Data on list activations, item interaction, and obstacle collisions are also automatically recorded and output to this file.

Representative Results

A project is currently underway to establish the validity of the CC-VRS in assessing cognitive, motor, and IADL function in young adults, older adults without neurological disease, and individuals with PD. Each participant completed the Tutorial, Basic, and Complex Scenarios using the same 150 m path and 5-item lists to allow for the comparison of performance across groups. Detailed cognitive and position data were utilized to establish informative summary metrics that distinguish CC-VRS performance between populations with known differences in cognitive, motor, and IADL function. Additional biomechanical and dual-tasking metrics were calculated to further characterize level of function across various domains (Table 1).

CC-VRS OutcomeDomain
Cognitive
Correct and incorrect itemsExecutive Function
List activations (number and duration)Working memory
Sale item recall (number correct)Declarative memory
Sale price comparison (success and duration)Processing speed
Cognitive-Motor
Trial durationGlobal (IADL) function
Stops (number and duration)Dual task interference
Gait speed in proximity of list itemsDual task interference
Collisions with avoidance obstaclesResponse inhibition
Motor
Velocity, step length, gait variabilityGait speed and quality
Turn velocity, turn durationTurn quality
Step width, symmetryPostural stability
Number of zero crossings in accelerationWalking fluidity
Reach and transport duration at selected itemsUpper extremity function

Table 1: CC-VRS outcome metrics. A non-exhaustive list of possible outcome metrics of the CC-VRS platform, designated as primarily cognitive, motor, or cognitive-motor in nature. These outcomes were developed on the basis of the task analysis used to design the CC-VRS as an ecologically valid assessment of IADL function. The domains captured by these outcomes represent the spectrum of single- and dual-task functions necessary for successful completion of grocery shopping and other IADLs. In contrast to existing neuropsychology and motor evaluations, the CC-VRS assesses these domains under conditions that more accurately reflect the complex demands of IADL environments within home and community settings. Abbreviations: CC-VRS = Cleveland Clinic-Virtual Reality Shopping; IADL = instrumental activity of daily living.

Figure 4 provides an overview of the Basic Scenario performance of one participant with PD. The participant's walking path through the store was compared to the ideal path of the navigational route, and the locations of correct shopping items were noted. Using positional data from the VR trackers, the participant's instantaneous velocity through the store was recorded and plotted. Adding the context of list activations and item selection provided insight into the participant's dual tasking ability and overall capacity to efficiently complete the IADL task.

Based on the results of preliminary analyses, the overall CC-VRS performance differed between healthy young adults and individuals with PD (Figure 5). Outcomes of total trial duration, number and duration of stops, and number and duration of list views during the task are promising metrics to differentiate between groups. Older adults and individuals with PD required more time to complete each scenario and spent more time stopped and activating the shopping list compared to healthy younger adults. Young adults displayed heightened dual-tasking capacity by simultaneously walking and activating the list, whereas individuals with PD more commonly activate the shopping list while stopped. Additional outcomes, including time spent searching for items, gait metrics, and the results of the cognitive demands in the Complex Scenario, are available for analysis.

In a separate CC-VRS usability study for individuals with PD, 10 participants completed the Simulator Sickness Questionnaire (SSQ)47,48 to evaluate the symptoms of VR sickness at baseline, immediately after completing the CC-VRS experience, and 30 min after completing the task. Developed in the context of flight simulations, the SSQ captures 16 common symptoms on a 4-point scale and has been adopted for use in VR applications. Individual symptom scores are combined and weighted to form subscores in the domains of nausea, oculomotor, and disorientation symptom clusters, in addition to a total score. Total SSQ scores range from 0 to 235.6.

Figure 6 displays the results of the SSQ completed at baseline (average total score 13.1 ± 16.7), immediately after CC-VRS (29.5 ± 27.9), and 30 min after CC-VRS (14.2 ± 15.6) for participants with PD (N = 10). In general, total SSQ scores for participants with PD were mild after CC-VRS, and the most commonly endorsed symptoms were general discomfort, fatigue, eye strain, difficulty focusing, and sweating. Notably, many of the participants reported mild symptoms at baseline. Nevertheless, 9/10 participants completed the full assessment, including the Tutorial, Basic, and Complex Scenarios, in an average of 29.0 ± 5.9 min. One was unable to tolerate the CC-VRS due to sickness. These data provide compelling evidence that the CC-VRS platform is well tolerated by most individuals with neurological disease. Collectively, the general lack of significant VR sickness symptoms reported suggests that coupling VR content with an omnidirectional treadmill is feasible and may address the VR locomotion problem for most individuals.

The 10 participants completing the usability study participated in a semistructured interview following their use of the CC-VRS. All participants endorsed that the study was their first time using VR and/or an omnidirectional treadmill. Summary remarks about the treadmill included the following:

Ease of treadmill adaption: Participants felt comfortable on the treadmill generally within a few minutes, as the walking mimicked overground stepping. Participants pointed out two aspects of gait that required adaptation: (1) The pull of the waist tracker back to the center of the treadmill during stopping and (2) taking slightly shorter steps due to the size of the treadmill platform.

Upper extremity support was stabilizing: The use of the circular handrail encompassing the treadmill provided an appropriate level of upper extremity support that aided in task completion.

Challenging physical and cognitive environment: Participants reported that their postural control was challenged while performing the shopping tasks. There was comfort in being harnessed, but the harness did not limit movement in any plane.

Realistic environment: The visual and auditory displays closely resembled a real grocery store and were impressive for VR-naïve users. Participants reported that the realism of the other shoppers and aisle obstacles motivated them to avoid collisions and that the navigational route was simple to follow.

Disorientation: Complaints of disorientation and sickness aligned with the individual SSQ scores. Some participants exhibited initial visuospatial challenges during the first several minutes of the CC-VRS that resulted in the individual coming in close approximation with the grocery shelves, which they felt created a feeling of disorientation.

Participants with PD from both aforementioned studies (N = 24) completed the System Usability Scale (SUS) following CC-VRS completion. The SUS is a 10-item questionnaire that measures ease-of-use, global satisfaction, and learnability of a system49,50. Scores range from 0 to 100, where 68 indicates average usability. Overall SUS scores between 72.6 and 78.8 correspond to a grade of "B" and scores greater than 78.8 earn an "A"51. Among 24 participants with PD who completed the CC-VRS platform (Tutorial, Basic, and Complex Sessions), the CC-VRS received an average score of 75.7 ± 18.9.

figure-representative results-9550
Figure 4: CC-VRS performance summary. (A) An individual with Parkinson's disease completing the Basic Scenario of the CC-VRS platform. (B) The navigational path and walking velocity of the participant as they complete the task. The blue squares represent an item that was on the shopping list and successfully retrieved. Embedded on the navigational guide line is a heatmap line representing the participant's instantaneous walking speed; baseline walking speed is calculated over the first 20 m straight-line walk. Any instantaneous speed less than 0.5x the baseline walking speed is red; instantaneous speed above 1.5x the aforementioned average speed is green. There is a linear transition from red to yellow to green between 0.5x and 1.5x of the average straight line walking speed. Walking speed over the course of the trial (C) and the number of list activations (D) are presented. Notably, this participant had 15 list views over the course of the trial, despite having only five items on the shopping list. Abbreviation: CC-VRS = Cleveland Clinic-Virtual Reality Shopping. Please click here to view a larger version of this figure.

figure-representative results-11078
Figure 5: CC-VRS of healthy young adult versus Parkinson's disease. The cumulative distance walked by a healthy young adult (A) and a participant with PD (B) during the performance of the Basic Scenario. In general, both participants walked approximately the same distance as they followed the navigational line. However, the participant with PD took substantially longer (410 s) than the young adult (350 s) to complete the scenario. The green bars represent the number and duration of a list activation during the task. The young adult viewed the list on seven occasions for a total of 73.1 s, while the participant with PD viewed the list on 16 occasions for a total of 94.3 s. The blue dots reflect a physical stop by the participant. Inspection of the young adult performance indicates that they had fewer overall stops and could simultaneously walk and view the list. Conversely, the participant with PD had 17 stops that each corresponded to a list view, suggesting they were unable to effectively dual-task (e.g., walk and view the list simultaneously). Abbreviations: CC-VRS = Cleveland Clinic-Virtual Reality Shopping; PD = Parkinson's disease. Please click here to view a larger version of this figure.

figure-representative results-12651
Figure 6: Symptom Experience following CC-VRS. A total of 10 participants with PD performed the CC-VRS as part of a usability study. Each participant completed the Simulator Sickness Questionnaire before, immediately after, and 30 min after finishing the CC-VRS experience. The SSQ captures 16 symptoms of VR sickness, with a maximum score of 235.6. Most participants with PD endorsed mild symptoms at baseline, with symptoms somewhat elevated immediately after the CC-VRS and returning to baseline levels within 30 min of completing the platform. The entire CC-VRS (Tutorial, Basic, and Complex Scenarios) took an average of 29 min to complete, and the average SSQ score upon completing the CC-VRS was 29.5 (in red). Abbreviations: CC-VRS = Cleveland Clinic-Virtual Reality Shopping; PD = Parkinson's disease; SSQ = Simulator Sickness Questionnaire. Please click here to view a larger version of this figure.

Discussion

The CC-VRS Platform, to date, appears to most effectively address the locomotion problem in VR by combining state-of-the-art VR content with an omnidirectional treadmill. A critical aspect of the seamlessly immersive environment of the CC-VRS is ensuring proper communication between the treadmill and VR software. Correct setup of all aspects of the VR system—including base stations, feet and waist trackers, and hand controllers—is imperative. If tracking is inconsistent or unreliable, adjustment of the orientation and placement of the base stations, or addition of another base station unit, is required. Proper coverage of the physical space provides stable synchronization between the VR hardware and omnidirectional treadmill and ensures the position and orientation data from the VR devices are complete, accurate, and precise52. Calibration of the omnidirectional treadmill is recommended at the start of every data collection session to ensure optimal responsivity while navigating the virtual environment.

Patient familiarization with the treadmill is critical prior to administering the CC-VRS. Although intuitive and simple to operate, the omnidirectional treadmill requires user familiarization that is best done prior to introduction of the VR headset and the resultant spatial orientation challenges. To meet the needs of the individual user and the goals of the present assessment, the following features are configurable for each CC-VRS scenario: 1) treadmill low or high maximum speed, 2) repetition of tutorial module, 3) route length through the store, and 4) number of items on shopping list. These modifications optimize the assessment for the cognitive, motor, and dual-tasking capabilities of a wide functional range of patients.

The lack of single-platform technology capable of standardizing IADL performance by using objective and quantitative outcomes that characterize cognitive and motor functioning represents a critical barrier in the early identification and effective treatment of age-related neurological diseases such as PD or Alzheimer's disease. Current methods estimating IADL function using self-report questionnaires, while easy to administer, are susceptible to bias. During self-report, older adults tend to over- or under-estimate IADL capabilities53. Similarly, informants completing IADL queries often misjudge capabilities due to the observers' misperceptions or knowledge gaps35.

An alternative to self-report and informant-rated questionnaires is performance-based IADL evaluation. Performance-based evaluations are typically completed by a trained Occupational or Physical Therapist. While a number of performance tests and guides are available, they are not conducive to integration into clinical care, often requiring ample time and specialized space and equipment not typically found in a primary care or neurology provider's office. One of the most widely used performance-based assessments, Direct Assessment of Functional Status (DAFS), requires approximately 40 min to administer, and its scoring is largely based on expert opinion of the test administrator. Although the DAFS is useful in staging Alzheimer's disease, it lacks sensitivity and does not detect IADL declines at the mild cognitive impairment stage24. Fusing the virtual and real worlds by combining VR with an omnidirectional treadmill provides an opportunity to capture IADL performance under complex cognitive conditions that better replicate real-world environments, potentially resulting in earlier diagnoses of neurological disease54.

The CC-VRS platform addresses the clinical gap by providing a standardized, systematic, objective, and quantitative approach to characterizing IADL capabilities in older adults and those with neurological disease. Based on preliminary usability testing and data, the Basic and Complex CC-VRS Scenarios can be completed altogether in less than 30 minutes. Similar to other immersive VR studies with PD18, the majority of people with PD experience mild motion sickness symptoms. From a usability perspective, the CC-VRS received an overall SUS rating of 75.7, corresponding to a letter grade "B" and falling between the "good" and "excellent" descriptor categories55. For comparison, a recent assessment of popular phone and tablet applications reports an average SUS score of 77.7 for the top 10 applications across all platforms56, including mobile applications such as The Weather Channel and YouTube. Comments from participants indicated that most users enjoyed the realism and ability to interact with the CC-VRS platform. Importantly, the participants felt challenged from a physical and cognitive aspect, indicating the design achieved its goal to create a dynamic platform that recreated a complex IADL experience.

We have previously demonstrated that technology can be successfully integrated into clinical workflows in the evaluation of patients with concussion57 and in a busy neurological service for patients with Multiple Sclerosis (MS)58. Further, the use of technology in the management of concussion improved outcomes and reduced costs59, while its use in the treatment of MS led to a 27% saving in time spent documenting in the electronic health record for each patient60. Considering the continual goal to reduce the cost of delivering care61 and that time spent documenting in the electronic health record is frequently cited for physician burnout62, the integration of the CC-VRS platform into clinical care is likely to provide a substantial value-add to hospital systems. Two projects are ongoing in which the CC-VRS platform is integrated into 1) a regional primary care family health center that primarily treats healthy older adults and 2) a specialized movement disorders clinic at the Cleveland Clinic.

The absence of an accurate and reliable physiological or digital biomarker for PD and Alzheimer's disease causes great difficulty in early diagnosis and in measuring disease progression63,64. The CC-VRS platform has the potential to provide a digital biomarker under a single technological platform that will enhance clinical care and could result in shorter and more efficient clinical trials by reducing reliance on subjective and highly variable clinical outcomes (e.g., Movement Disorder Society - Unified Parkinson's disease Rating Scale motor portion (MDS-UPDRS III)). The evaluation of motor and cognitive function in the field of clinical neurology has not advanced dramatically over the past three decades in terms of the assessment of individuals with PD and the associated cardinal motor symptoms, let alone cognitive or dual-task problems. The most celebrated advancement in the assessment of individuals with PD is the revision of the subjective clinical rating scale (MDS-UPDRS III). Importantly, we do not believe that the CC-VRS will supplant the MDS-UPDRS III. Rather, we believe its greatest value may be realized in primary care practices by providing a standardized and objective approach to the quantification of IADLs. While it is premature to believe the CC-VRS in its current form is a prodromal marker of neurological disease, results could be used to raise a "red" or "yellow" flag in terms of neurological functioning that may trigger a consult by a movement disorders, neuropsychology, or geriatric specialist. In terms of its use in PD clinical care, it is anticipated that the CC-VRS could be utilized in the titration of medication or in the eventual programming of deep brain stimulation devices. Both the Primary Care and PD-specific use cases are currently in the pilot phase. By truly immersing the user in a realistic environment and measuring meaningful and important aspects of cognitive and motor function, the CC-VRS represents an initial step in the creation of a potentially effective and scalable digital biomarker for neurological disease.

The field of clinical neurology, movement disorders in particular, is filled with examples of technology developed to quantify a single, isolated PD symptom via accelerometer or other sensor technologies65,66,67,68,69. To our knowledge, none of these approaches, other than our balance70,71,72,73 and tremor applications74, have been integrated into routine PD clinical care. Previous technology oftentimes is valid and reliable; however, the focus has been on technology development with little regard to feasibility of clinical integration75,76. Patients, providers, hospitals, and regulatory bodies are increasingly interested in outcome measures that quantify changes in meaningful daily actions77,78,79,80. The clinical integration of precise and meaningful measures of neurological symptoms and IADL performance is necessary to systematically evaluate the overall effectiveness of an intervention or determine the potential of an intervention to slow disease progression. The development of a standardized approach to IADL assessment suitable for routine clinical use is appealing to facilitate comprehensive understanding and treatment of neurological disease on meaningful activities.

The CC-VRS approach to the evaluation of IADL performance to aid in the diagnosis and management of neurological disease has the potential to transform healthcare through early diagnosis and more precise tracking of disease progression. However, it is fully acknowledged that the system is not without limitation. The cost of the omnidirectional treadmill is substantial and may serve as a barrier for widespread adoption sans systematic health economics studies to identify the potential "break even" point between cost of the assessment relative to the value of early diagnosis or more precise tracking of disease progression. Notably, gaps in acquiring PD patient-centric outcomes with technology were highlighted by the National Institute of Neurological Disorders and Stroke PD Conference78, MDS Task Force on Technology77, and FDA. They called for technology that measures meaningful PD activities and integration of these outcomes into clinical care. We are currently evaluating the integration of the CC-VRS into a primary care setting and a movement disorders center at the Cleveland Clinic; these deployments will utilize more affordable omnidirectional treadmills. Successful collection of data does require an initial investment of time by the clinician to learn how to set up and operate the system. Ongoing clinical pilots will better inform the amount of training required to become a proficient user. One could imagine a model in which a technician is employed to operate the system, and patients complete the shopping tasks rather than sitting in a waiting room before an appointment. Those data could then be instantaneously integrated into the electronic health record prior to seeing their provider. These types of applications have the potential to become the waiting room of the future for patients.

Acknowledgements

This study was sponsored by the Michael J. Fox Foundation for Parkinson's Research (MJFF-020020) and the Edward and Barbara Bell Family Chair. We thank Elm Park Labs (Detroit, MI) for assistance in building the VR environment and linking with the omnidirectional treadmill. We also thank Evelyn Thoman and Brittney Moser for their assistance with project development and execution.

Materials

NameCompanyCatalog NumberComments
CleanboxCleanboxUV hygienic cleaning system used for disenfecting the VR headset
Desktop PCDellHigh-end gaming desktop
Infinadeck Omnidirectional TreadmillInfinadeckOmnidirectional treadmill allows you walk in any direction
Safety Harness YmachrayStandard saftey harness to prevent the patient from falling
Valve Index Base Stations x3ValveTracking of the headset/controllers and trackers
Valve Index Controllers (one set of 2)ValveHand controllers to interact with the digital content
Valve Index VR headsetValveVR headset
Vive tracker 3.0 x3HTCPlaced on feet and waist to track position and control movement of treadmill
Vive tracker strapsSkywin VRSecures the Vive tracker around the waist
Zip tiesUsed to affix Vive trackers to shoelaces

References

  1. Grand Challenges for Engineering. National Academy of Sciences Available from: https://16mhpx3atvadrnpip2kwi9or-wpengine.netdna-ssl.com/wp-content/uploads/2016/12/GrandChallenges.pdf (2008)
  2. Turso-Finnich, T., Jensen, R. O., Jensen, L. X., Konge, L., Thinggaard, E. Virtual reality head-mounted displays in medical education-a systematic review. Simulation in Healthcare. , (2022).
  3. Chen, T., et al. Virtual reality as a learning tool in spinal anatomy and surgical techniques. North American Spine Society Journal. 6, 100063 (2021).
  4. Gold, J. I., SooHoo, M., Laikin, A. M., Lane, A. S., Klein, M. J. Effect of an immersive virtual reality intervention on pain and anxiety associated with peripheral intravenous catheter placement in the pediatric setting: a randomized clinical trial. JAMA Network Open. 4 (8), 2122569 (2021).
  5. Huang, Q., Lin, J., Han, R., Peng, C., Huang, A. Using virtual reality exposure therapy in pain management: a systematic review and meta-analysis of randomized controlled trials. Value Health. 25 (2), 288-301 (2022).
  6. Groninger, H., Stewart, D., Wesley, D., Cowgill, J., Mete, M. Virtual reality for management of cancer pain: Study rationale and design. Contemporary Clinical Trials Communications. 26, 100895 (2022).
  7. Zhang, B., Li, D., Liu, Y., Wang, J., Xiao, Q. Virtual reality for limb motor function, balance, gait, cognition and daily function of stroke patients: A systematic review and meta-analysis. Journal of Advanced Nursing. 77 (8), 3255-3273 (2021).
  8. Saredakis, D., et al. Factors Associated with virtual reality sickness in head-mounted displays: a systematic review and meta-analysis. Frontiers in Human Neuroscience. 14, 96 (2020).
  9. Kim, H. K., Park, J., Choi, Y., Choe, M. Virtual reality sickness questionnaire (VRSQ): Motion sickness measurement index in a virtual reality environment. Applied Ergonomics. 69, 66-73 (2018).
  10. Cobb, S. V. G., Nichols, S., Ramsey, A., Wilson, J. R. Virtual reality-induced symptoms and effects (VRISE). Presence-Teleoperators and Virtual Environments. 8, 169-186 (1999).
  11. Kennedy, R., Lane, N., Lilienthal, M., Berbaum, K., Hettinger, L. Profile analysis of simulator sickness symptoms: application to virtual environment systems. Presence-Teleoperators and Virtual Environments. 1 (3), 295-301 (1992).
  12. Duzmanska, N., Strojny, P., Strojny, A. Can simulator sickness be avoided? a review on temporal aspects of simulator sickness. Frontiers in Psychology. 9, 2132 (2018).
  13. Reason, J. T. Motion sickness adaptation: a neural mismatch model. Journal of the Royal Society of Medicine. 71 (11), 819-829 (1978).
  14. Chance, S. S., Gaunet, F., Beall, A. C., Loomis, J. M. Locomotion mode affects the updating of objects encountered during travel: the contribution of vestibular and proprioceptive inputs to path integration. Presence Teleoperators & Virtual Environments. 7 (2), 168-178 (1998).
  15. Waller, D., Bachmann, E., Hodgson, E., Beall, A. C. The HIVE: a huge immersive virtual environment for research in spatial cognition. Behavior Research Methods. 39 (4), 835-843 (2007).
  16. Loomis, J. M., Blascovich, J. J., Beall, A. C. Immersive virtual environment technology as a basic research tool in psychology. Behavior Research Methods, Instruments, & Computers. 31 (4), 557-564 (1999).
  17. Mirelman, A., et al. Addition of a non-immersive virtual reality component to treadmill training to reduce fall risk in older adults (V-TIME): a randomised controlled trial. Lancet. 388 (10050), 1170-1182 (2016).
  18. Kim, A., Darakjian, N., Finley, J. M. Walking in fully immersive virtual environments: an evaluation of potential adverse effects in older adults and individuals with Parkinson's disease. Journal of NeuroEngineering and Rehabilitation. 14 (1), 16 (2017).
  19. Pelosin, E., et al. Motor-cognitive treadmill training with virtual reality in Parkinson's disease: the effect of training duration. Frontiers in Aging Neuroscience. 13, 753381 (2021).
  20. Darken, R. P., Cockayne, W. R., Carmein, D. The omni-directional treadmill: A locomotion device for virtual worlds. Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology. , 213-221 (1997).
  21. Guo, H. J., Sapra, A. . Instrumental Activity of Daily Living. , (2021).
  22. Darweesh, S. K., et al. Trajectories of prediagnostic functioning in Parkinson's disease. Brain. 140 (2), 429-441 (2017).
  23. Foubert-Samier, A., et al. Cognitive and functional changes in prediagnostic phase of Parkinson disease: A population-based study. Parkinsonism & Related Disorders. 79, 40-46 (2020).
  24. Marshall, G. A., Amariglio, R. E., Sperling, R. A., Rentz, D. M. Activities of daily living: where do they fit in the diagnosis of Alzheimer's disease. Neurodegenerative Disease Management. 2 (5), 483-491 (2012).
  25. Sikkes, S. A., et al. Assessment of instrumental activities of daily living in dementia: diagnostic value of the Amsterdam Instrumental Activities of Daily Living Questionnaire. Journal of Geriatric Psychiatry and Neurology. 26 (4), 244-250 (2013).
  26. American Occupational Therapy Association. Occupational therapy practice framework: domain and process. American Journal of Occupational Therapy. 56 (6), 609-639 (2002).
  27. MacPherson, S. E. Definition: Dual-tasking and multitasking. Cortex. 106, 313-314 (2018).
  28. O'Shea, S., Morris, M. E., Iansek, R. Dual task interference during gait in people with Parkinson disease: effects of motor versus cognitive secondary tasks. Physical Therapy. 82 (9), 888-897 (2002).
  29. Romero-Ayuso, D., et al. Assessment of cognitive instrumental activities of daily living: a systematic review. Disability and Rehabilitation. 43 (10), 1342-1358 (2019).
  30. Goetz, C. G., et al. Movement Disorder Society-sponsored revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS): scale presentation and clinimetric testing results. Movement Disorders. 23 (15), 2129-2170 (2008).
  31. Perlmutter, J. S. Assessment of Parkinson disease manifestations. Current Protocols in Neuroscience. , 1382-1387 (2009).
  32. Palmer, J. L., et al. Unified Parkinson's Disease Rating Scale-Motor Exam: inter-rater reliability of advanced practice nurse and neurologist assessments). Journal of Advanced Nursing. 66 (6), 1382-1387 (2010).
  33. Neisser, U. On "Social Knowing". Personality and Social Psychology Bulletin. 6 (4), 601-604 (1980).
  34. Neisser, U. . Memory Observed: Remembering in Natural Contexts. , (1982).
  35. Jekel, K., et al. Mild cognitive impairment and deficits in instrumental activities of daily living: a systematic review. Alzheimer's Research & Therapy. 7 (1), 17 (2015).
  36. Chisholm, D., Toto, P., Raina, K., Holm, M., Rogers, J. Evaluating capacity to live independently and safely in the community: Performance assessment of self-care skills. British Journal of Occupational Therapy. 77 (2), 59-63 (2014).
  37. Cipresso, P., et al. Virtual multiple errands test (VMET): a virtual reality-based tool to detect early executive functions deficit in Parkinson's disease. Frontiers in Behavioral Neuroscience. 8, 405 (2014).
  38. Owsley, C., Sloane, M., McGwin, G., Ball, K. Timed instrumental activities of daily living tasks: relationship to cognitive function and everyday performance assessments in older adults. Gerontology. 48 (4), 254-265 (2002).
  39. Brennan, L., et al. The Penn Parkinson's Daily Activities Questionnaire-15: Psychometric properties of a brief assessment of cognitive instrumental activities of daily living in Parkinson's disease. Parkinsonism & Related Disorders. 25, 21-26 (2016).
  40. Arlati, S., et al. Acceptance and usability of immersive virtual reality in older adults with objective and subjective cognitive decline. Journal of Alzheimer's Disease. 80 (3), 1025-1038 (2021).
  41. Porffy, L. A., et al. A novel virtual reality assessment of functional cognition: validation study. Journal of Medical Internet Research. 24 (1), 27641 (2022).
  42. Swanson, C. W., Fling, B. W. Discriminative mobility characteristics between neurotypical young, middle-aged, and older adults using wireless inertial sensors. Sensors. 21 (19), 6644 (2021).
  43. Yeh, T. T., Liang, P. J., Lee, S. C. Differences in walking-to-turning characteristics between older adult fallers and nonfallers: a prospective and observational study using wearable inertial sensors. International Journal of Rehabilitation Research. 45 (1), 53-57 (2022).
  44. Zach, H., et al. Identifying freezing of gait in Parkinson's disease during freezing provoking tasks using waist-mounted accelerometry. Parkinsonism & Related Disorders. 21 (11), 1362-1366 (2015).
  45. Bhatt, H., Pieruccini-Faria, F., Almeida, Q. J. Dynamics of turning sharpness influences freezing of gait in Parkinson's disease. Parkinsonism & Related Disorders. 19 (2), 181-185 (2013).
  46. Hoops, S., et al. Validity of the MoCA and MMSE in the detection of MCI and dementia in Parkinson disease. Neurology. 73 (21), 1738-1745 (2009).
  47. Bruck, S., Watters, P. A. Estimating cybersickness of simulated motion using the simulator sickness questionnaire (SSQ): A controlled study. Proceedings of the 2009 Sixth International Conference on Computer Graphics, Imaging and Visualization; Institute of Electrical and Electronics Engineers (IEEE). , 486-488 (2009).
  48. Kennedy, R. S., Lane, N. E., Berbaum, K. S., Lilienthal, M. G. Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. The International Journal of Aviation Psychology. 3 (3), 203-220 (1993).
  49. Brooke, S. . Usability Evaluation in Industry. , 189-194 (1996).
  50. Lewis, J. R., Sauro, J. The factor structure of the system usability scale. Human Centered Design. , 94-103 (2009).
  51. Sauro, J., Lewis, J. R. . Quantifying the User Experience: Practical Statistics for User Research. 2nd ed. , (2016).
  52. Niehorster, D. C., Li, L., Lappe, M. The accuracy and precision of position and orientation tracking in the HTC Vive virtual reality system for scientific research. i-Perception. 8 (3), 2041669517708205 (2017).
  53. Suchy, Y., Kraybill, M. L., Franchow, E. Instrumental activities of daily living among community-dwelling older adults: discrepancies between self-report and performance are mediated by cognitive reserve. Journal of Clinical and Experimental Neuropsychology. 33 (1), 92-100 (2011).
  54. Reppermund, S., et al. Impairment in instrumental activities of daily living with high cognitive demand is an early marker of mild cognitive impairment: the Sydney memory and ageing study. Psychological Medicine. 43 (11), 2437-2445 (2013).
  55. Bangor, A., Kortum, P. T., Miller, J. T. An empirical evaluation of the system usability scale. International Journal of Human-Computer Interaction. 24 (6), 574-594 (2008).
  56. Kortum, P., Sorber, M. Measuring the usability of mobile applications for phones and tablets. International Journal of Human-Computer Interaction. 31 (8), 518-529 (2015).
  57. Alberts, J. L., et al. Development and implementation of a multi-disciplinary technology enhanced care pathway for youth and adults with concussion. Journal of Visualized Experiments. (143), e58962 (2019).
  58. Rhodes, J. K., et al. Multiple Sclerosis performance test: technical development and usability. Advances in Therapy. 36 (7), 1741-1755 (2019).
  59. Alberts, J. L., et al. A technology-enabled concussion care pathway reduces costs and enhances care. Physical Therapy. 100 (1), 136-148 (2020).
  60. Macaron, G., et al. Technology-enabled assessments to enhance multiple sclerosis clinical care and research. Neurology Clinical Practice. 10 (3), 222-231 (2020).
  61. Porter, M. E. What is value in health care. The New England Journal of Medicine. 363 (26), 2477-2481 (2010).
  62. Sutton, J. M., Ash, S. R., Al Makki, A., Kalakeche, R. A. A daily hospital progress note that increases physician usability of the electronic health record by facilitating a problem-oriented approach to the patient and reducing physician clerical burden. The Permanente Journal. 23, (2019).
  63. Maetzler, W., et al. Modernizing daily function assessment in Parkinson's disease using capacity, perception, and performance measures. Movement Disorders. 36 (1), 76-82 (2021).
  64. Stephenson, D., Badawy, R., Mathur, S., Tome, M., Rochester, L. Digital progression biomarkers as novel endpoints in clinical trials: a multistakeholder perspective. Journal of Parkinson's Disease. 11, 103-109 (2021).
  65. Lu, M., et al. Vision-based estimation of MDS-UPDRS Gait scores for assessing Parkinson's Disease motor severity. Medical Image Computing and Computer-Assisted Intervention. 12263, 637-647 (2020).
  66. Hobert, M. A., et al. Progressive gait deficits in Parkinson's disease: a wearable-based biannual 5-year prospective study. Frontiers in Aging Neuroscience. 11, 22 (2019).
  67. Thorp, J. E., Adamczyk, P. G., Ploeg, H. L., Pickett, K. A. Monitoring motor symptoms during activities of daily living in individuals with Parkinson's disease. Frontiers in Neurology. 9, 1036 (2018).
  68. Shawen, N., et al. Role of data measurement characteristics in the accurate detection of Parkinson's disease symptoms using wearable sensors. Journal of NeuroEngineering and Rehabilitation. 17 (1), 52 (2020).
  69. Lu, R., et al. Evaluation of wearable sensor devices in Parkinson's disease: a review of current status and future prospects. Parkinsons Disease. 2020, 4693019 (2020).
  70. Ozinga, S. J., Alberts, J. L. Quantification of postural stability in older adults using mobile technology. Experimental Brain Research. 232 (12), 3861-3872 (2014).
  71. Ozinga, S. J., et al. Three-dimensional evaluation of postural stability in Parkinson's disease with mobile technology. NeuroRehabilitation. 41 (1), 211-218 (2017).
  72. Ozinga, S. J., Linder, S. M., Alberts, J. L. Use of mobile device accelerometry to enhance evaluation of postural instability in Parkinson disease. Archives of Physical Medicine and Rehabilitation. 98 (4), 649-658 (2017).
  73. Ozinga, S. J., Machado, A. G., Miller Koop, M., Rosenfeldt, A. B., Alberts, J. L. Objective assessment of postural stability in Parkinson's disease using mobile technology. Movement Disorders. 30 (9), 1214-1221 (2015).
  74. Maldonado-Naranjo, A., Koop, M. M., Hogue, O., Alberts, J., Machado, A. Kinematic metrics from a wireless stylus quantify tremor and bradykinesia in Parkinson's disease. Parkinson's Disease. 2019, 6850478 (2019).
  75. Lingaiah, A., et al. Improving anxiety in Parkinson's disease: A cautionary tale about mobile health technologies. Parkinsonism & Related Disorders. 73, 50-51 (2020).
  76. di Biase, L., et al. Quantitative analysis of bradykinesia and rigidity in Parkinson's disease. Frontiers in Neurology. 9, 121 (2018).
  77. Espay, A. J., et al. Technology in Parkinson's disease: Challenges and opportunities. Movement Disorders. 31 (9), 1272-1282 (2016).
  78. Sieber, B. A., et al. Prioritized research recommendations from the National Institute of Neurological Disorders and Stroke Parkinson's Disease 2014 conference. Annals of Neurology. 76 (4), 469-472 (2014).
  79. van Uem, J. M., et al. Health-related quality of life in patients with Parkinson's disease--A systematic review based on the ICF model. Neuroscience & Biobehavioral Reviews. 61, 26-34 (2016).
  80. Papadopoulos, E., Buracchio, T. Drug Development Tool (DDT) COA #000142. U.S. Food & Drug Administration. , (2021).

Explore More Articles

Virtual RealityInstrumental Activities Of Daily LivingOmnidirectional TreadmillMotion SicknessParkinson s DiseaseAlzheimer s DiseaseObjective AssessmentCalibrationBody TrackingHarness

This article has been published

Video Coming Soon

JoVE Logo

Privacy

Terms of Use

Policies

Contact Us

Recommend to library

JoVE NEWSLETTERS

Research

Education

Authors

Librarians

ABOUT JoVE

Copyright © 2024 MyJoVE Corporation. All rights reserved