Authors
Contact Us

Sign In

A subscription to JoVE is required to view this content. Sign in or start your free trial.

In This Article

  • Summary
  • Abstract
  • Introduction
  • Protocol
  • Results
  • Discussion
  • Disclosures
  • Acknowledgements
  • Materials
  • References
  • Reprints and Permissions

Summary

Wearable technology has low cost and offers convenient monitoring of physiological data. However, the accuracy and reliability of these devices require cautious assessment to ensure their effectiveness and safety for users. This report describes the validation process of a commercial smartwatch in monitoring physiological data and physical activity.

Abstract

This study aims to validate the accuracy of low-cost fitness smartwatches by comparing their data with gold-standard measurements for cardiovascular and physical activity parameters. The study enrolled 50 subjects, 26 undergoing validation testing for heart rate, blood oxygen saturation (SpO2), and sleep data against polysomnography (PSG). Additionally, 24 subjects participated in the 3-Minute Walk Test (3MWT) and Stairs Climbing (SC), with step counts validated against manual video calculations. Results showed no significant difference between the device's measurements and gold standard values for shallow sleep, deep sleep, REM time, mean heart rate, minimum heart rate, and SpO2. However, the device significantly underestimated manually counted steps (p = 0.009 (3MWT); p = 0.012 (SC)), total sleep duration (p = 0.004), and wake time (p = 8.94 × 10-8) while overestimating maximum heart rate (p = 0.011). These findings highlight the importance of accurate validation and interpretation of wearable device data in clinical contexts. Given these limitations, excluding the device's readings in future analyses is recommended to maintain data reliability and research integrity. This study underscores the need for ongoing validation and improvement of wearable technology to ensure its reliability and effectiveness in healthcare.

Introduction

Wearable technology has grown in popularity, becoming commonplace in various parts of daily life1. These technologies, equipped with sensors and algorithms, have transformed how physiological parameters are monitored and interpreted, providing users with health information, tracking workouts, and allowing users to have a healthier lifestyle. The integration of artificial intelligence and pattern recognition, combined with increasingly popular features like virtual and augmented reality features, not only enhances the functionality of wearable devices but also enables advanced personalized data analysis and more engaging user experience2,3. As wearable devices become more integrated into everyday routines, understanding their accuracy and reliability becomes paramount, especially in industries such as sports performance monitoring and healthcare.

Wearables offer the benefit of real-time patient data logging for chronic illnesses, providing invaluable insights for both patients and healthcare professionals. Metrics such as physical activity, conversation timelines, and sleep duration can help assess the severity of diseases such as depression, sleep apnea, and Parkinson's disease4,5,6. Emerging technologies, such as electronic stamps that monitor glucose, heart rate, and temperature, are undergoing clinical testing, further expanding the possibilities of advanced wearables in healtchcare7. Furthermore, athletes and fitness enthusiasts utilize wearable technology to optimize training regimens, track performance metrics, and prevent injuries8,9. The integration of regular evaluation of physiological parameters, activity data, and functionalities like electrocardiogram recording has sparked interest regarding wearables' potential to aid in the diagnosis and tracking of cardiovascular diseases8,10. Wearable gadgets have shown promise in enhancing workplace safety, improving lifestyle management, and instilling healthier habits among users11.

Fitness smartwatch bands, such as the Xiaomi Mi Band, a well-known low-cost wearable technology and one of the most widely used smartwatches globally12, have been applied in a variety of applications, including medicine, sports, and education13. For example, research has utilized fitness smartwatches for monitoring the physiological parameters of older individuals14,15, analyzing student behavior during STEM education13, and assessing physical activity in individuals16,17. Despite their extensive use, there are ongoing questions concerning the precision and reliability of the measures provided by these devices. Numerous investigations have attempted to validate the measurements obtained from wearable technology12,18,19,20. Previous studies investigated the accuracy of step counting and the functionality of sleep monitoring, providing insights into the devices' performance in a wide range of settings and user demographics. However, the existing research indicates significant gaps in knowledge, such as contradictory findings across studies and insufficient validation against gold standard methodologies such as polysomnography21,22,23.

The purpose of this study is to validate the data obtained from fitness smartwatches and address the reliability of fitness smartwatches. The goal of evaluating the wearable device's sleep-tracking accuracy and other relevant metrics such as heart rate, blood oxygen level, and steps is to provide significant insights into the device's suitability for clinical and research applications that require a high level of accuracy and precision. This is intended to contribute to the growing body of research on wearable technology validation through rigorous methodology and solid statistical analysis, ultimately improving the ability to employ these tools for better health outcomes and well-being.

Protocol

The study protocol is approved by the Universiti Malaya Medical Centre (UMMC) Ethics Review Board (MREC No: 2021325-9983). The validation study of fitness smartwatch measurements is divided into three parts: (1) validation of sleep measurements against the gold standard using a PSG machine, (2) validation of step measurements by comparing manual calculations from video recordings, and (3) data analysis of the validation tests.

1. Validation of sleep measurements against polysomnography (PSG)

NOTE: The study is conducted in a controlled sleep laboratory to minimize external disruptions. Prior to the test, all participants were briefed in depth about the study and were provided with the study information sheet before written informed consent was obtained. All information gathered is kept anonymous.

  1. Ensure participants arrive 1 h before the test starts for necessary preparation.
  2. Participants wear the fitness smartwatch on their non-dominant wrist.
  3. Attach participants to the apparatus of the PSG machine according to the manufacturer's manual24.
    NOTE: Use standardized placement of PSG electrodes as per the manufacturer's manual to ensure minimal discomfort to participants during sleep.
  4. Take two measurements of blood oxygen saturation level (SpO2) using the fitness smartwatch in a supine position.
  5. Ensure participants sleep for a maximum 6 h.
    NOTE: Technical personnel monitor the PSG machine throughout the sleep duration.
  6. Take two measurements of SpO2 using the fitness smartwatch in a supine position after the participant is awake.
  7. Export sleep trends from both PSG machine and fitness smartwatch, i.e., deep sleep duration, light sleep duration, rapid eye movements (REM) time and sleep duration, heart rate (minimum, mean, maximum heart rate), and SpO2, into a .csv file. Refer to each device's manufacturer manual on how to export the data24,25.
  8. Proceed to section 3 for data analysis.

2. Validation of fitness smartwatch step measurements with manual calculation from video recordings

NOTE: Prior to the test, all participants are briefed in depth about the study and are provided with the study information sheet before written informed consent is obtained. All information gathered is kept anonymous. Only participants with good health conditions and no impairment in mobility are chosen for this validation test.

  1. Ensure participants wear the fitness smartwatch on their non-dominant wrist and hold a smartphone for footstep video recording purposes on the other hand.
  2. Set the step count on the fitness smartwatch to zero. Start recording on the smartphone and have the participant perform the step test as indicated in step 2.3. Stop and save the recording. Record the number of steps measured in the fitness smartwatch.
  3. Ensure participant completes two types of step tests (i) 3MWT and (ii) SC tests.
    1. 3MWT: Ensure the participant walks at a normal and steady pace for 3 min on a flat surface.
    2. SC test: Have the participant take two flights of stairs in ascending and descending order at a normal and steady pace.
  4. Replay the recording and calculate the number of steps from the video recording. Record the data into a .csv file. Export the step counts from the fitness smartwatch into the same file. Refer to the fitness smartwatch manufacturer's manual on how to export the data25.
  5. Proceed to section 3 for data analysis.

3. Data analysis of the validation tests

NOTE: In this protocol, the scipy.stats package in Python was used to perform all analysis tests except the Cohen d test, where the pingouin package was utilized instead. The code is in Supplementary File 1, where each step is mentioned in the comment line.

  1. Import package required into Python's workspace.
  2. Import measurement data into the Python's workspace. Replace the filename measurement_data.csv with a filename of choice.
  3. Remove any empty values using the dropna() function for each measurement data. Then, perform the Shapiro-Wilk normality test to determine if the data has a normal distribution using the scipy.stats shapiro package in Python.
  4. Perform paired sample t-test using the scipy.stats ttest_rel package in Python if data has a normal distribution, else, perform Mann-Whitney Wilcoxon using the scipy.stats wilcoxon package in Python. If the result has a p-value < 0.05, it means there is a significant difference between the measurement data from both instruments. In the code, replace measure1_toolA and measure1_toolB with the respective column names of choice.
  5. Measure the magnitude of the difference between instruments via effect size using the Cohen d method. An output of 0.2 indicates a small effect, 0.5 a moderate effect, and 0.8 a large effect.
  6. Employ Bland-Altman analysis to assess the agreement between two instruments, such as mean or median difference, standard deviation (SD), and 95% CI of the bias. Plot the Bland-Altman analysis for easy visualization of the output using the matplotlib.pyplot package in Python.
  7. Repeat step 3.5 until step 3.7 for each pair of the measurement data from both instruments.

Results

Table 1 shows significant differences (p-value < 0.05) between data from PSG and the fitness smartwatch during the first part of the validation test. The fitness smartwatch overestimated wake time (p < 0.001), underestimated sleep duration (p = 0.004), and reported a higher maximal heart rate (p = 0.001). However, no significant differences were found between the PSG machine and the fitness smartwatch in the following measurements: shallow sleep time, deep sleep time, REM time, mean heart rate, m...

Discussion

Several limitations of the fitness smartwatch were identified based on the mixed results in the comparison analysis. The high variability in the smartwatch's measurements may stem from its reliance on movement and heart rate data rather than more detailed measures, such as the electroencephalogram (EEG) used in PSG. Deep sleep and REM time showed better agreement, which suggests that the fitness smartwatch may be suitable for general sleep monitoring. However, it may not be reliable for clinical diagnosis, such as detail...

Disclosures

The authors declare that the research was conducted in the absence of any commercial or financial compensation, sponsorship, or any relationships with the device manufacturer that could be construed as a potential conflict of interest. The opinions and findings expressed in this article are the author's own and are based solely on their experience with the product from this study.

Acknowledgements

We thank the neurotechnologist at the Neurology Lab, University of Malaya Medical Centre, for the help and support in conducting the test for this study. This work was supported by the UM Research Center (IIRG001C-2021IISS).

Materials

NameCompanyCatalog NumberComments
Sleep Diagnostic System Natus Neurology & CompumedicsReferred in manuscript as PSG machine
Xiaomi Mi Band 6XiaomiReferred in manuscript as fitness smartwatch

References

  1. Sergueeva, K., Shaw, N., Lee, S. H. Understanding the barriers and factors associated with consumer adoption of wearable technology devices in managing personal health. Can J Adm Sci. 37 (1), 45-60 (2020).
  2. Xu, K., Lu, Y., Takei, K. Multifunctional skin-inspired flexible sensor systems for wearable electronics. Adv Mater Technol. 4 (3), 25 (2019).
  3. Ates, H. C. et al. End-to-end design of wearable sensors. Nat Rev Mater. 7 (11), 887-907 (2022).
  4. Harrington, J., Schramm, P. J., Davies, C. R., Lee-Chiong, T. L., Jr. An electrocardiogram-based analysis evaluating sleep quality in patients with obstructive sleep apnea. Sleep Breath. 17 (3), 1071-1078 (2013).
  5. Mccall, W. V. A rest-activity biomarker to predict response to SSRIs in major depressive disorder. J Psychiatr Res. 64, 19-22 (2015).
  6. Arora, S. et al. High accuracy discrimination of Parkinson's disease participants from healthy controls using smartphones. in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Florence, Italy (2014).
  7. Perry, T. Temporary tattoos can replace today's biomedical sensors. IEEE Spectr. 34, 84-85 (2015).
  8. Lehra, C. et al. Wearable applications in rugby for performance quantification and player health assessment: A brief review. IJKSS. 10 (2), 10 (2022).
  9. Lacey, A., Whyte, E., O'keeffe, S., O'connor, S., Moran, K. A qualitative examination of the factors affecting the adoption of injury focused wearable technologies in recreational runners. PloS One. 17 (7), 22 (2022).
  10. Binyamin, S. S. Hoque, M. R. Understanding the drivers of wearable health monitoring technology: An extension of the unified theory of acceptance and use of technology. Sustainability. 12 (22), 20 (2020).
  11. Rao, P., Seshadri, D. R., Hsu, J. J. Current and potential applications of wearables in sports cardiology. Curr Treat Options Cardiovasc Med. 23 (10), 65 (2021).
  12. Casado-Robles, C., Mayorga-Vega, D., Guijarro-Romero, S., Viciana, J. Validity of the xiaomi mi band 2, 3, 4 and 5 wristbands for assessing physical activity in 12-to-18-year-old adolescents under unstructured free-living conditions. Fit-person study. J Sports Sci Med. 22 (2), 196-211 (2023).
  13. Shapovalov, Y. B. et al. Using of personal smart tools in STEM education. Proceedings of the 1st Symposium on Advances in Educational Technology. 2, 192-207 (2020).
  14. Concheiro-Moscoso, P. et al. Use of the Xiaomi mi band for sleep monitoring and its influence on the daily life of older people living in a nursing home. Digit Health. 8, 20552076221121162 (2022).
  15. Ardiyanto, Y., Adi, K., Putra, K. T., Utomo, P. Monitoring system for elderly health care using smart band, Raspberry Pi, and Node-Red. 2022 2nd International Conference on Electronic and Electrical Engineering and Intelligent System (ICE3IS). Yogyakarta, Indonesia (2022).
  16. Viciana, J., Casado-Robles, C., Guijarro-Romero, S., Mayorga-Vega, D. Are wrist-worn activity trackers and mobile applications valid for assessing physical activity in high school students? Wearfit study. J Sports Sci Med. 21 (3), 356-375 (2022).
  17. Hao, Y., Ma, X. -K., Zhu, Z., Cao, Z. -B. Validity of wrist-wearable activity devices for estimating physical activity in adolescents: Comparative study. JMIR Mhealth Uhealth. 9 (1), e18320 (2021).
  18. Ameen, M. S., Cheung, L. M., Hauser, T., Hahn, M. A., Schabus, M. About the accuracy and problems of consumer devices in the assessment of sleep. Sensors. 19 (19), 4160 (2019).
  19. Concheiro-Moscoso, P. et al. Quality of sleep data validation from the xiaomi mi band 5 against polysomnography: Comparison study. J Med Internet Res. 25, e42073 (2023).
  20. Topalidis, P. et al. Evaluation of a low-cost commercial actigraph and its potential use in detecting cultural variations in physical activity and sleep. Sensors. 21 (11), 18 (2021).
  21. Xu, H., Zheng, X., Jia, W., Yin, S. Chromatography/mass spectrometry-based biomarkers in the field of obstructive sleep apnea. Medicine. 94 (40), e1541 (2015).
  22. Michaelson, P. G., Allan, P., Chaney, J., Mair, E. A. Validations of a portable home sleep study with twelve-lead polysomnography: Comparisons and insights into a variable gold standard. Ann Otol Rhinol Laryngol. 115 (11), 802-809 (2006).
  23. Berry, R. B. et al. Rules for scoring respiratory events in sleep: Update of the 2007 AASM manual for the scoring of sleep and associated events. J Clin Sleep Med. 08 (05), 597-619 (2012).
  24. Natus Neurology Incorporated. Nicoletone information for use. At <https://natus.com/> (2024).
  25. Xiaomi Communications. Mi smart band 6 user manual. At <https://www.manualslib.com/manual/2629650/Xiaomi-Mi-Smart-Band-6.html> (2021).
  26. De Zambotti, M., Cellini, N., Goldstone, A., Colrain, I. M., Baker, F. C. Wearable sleep technology in clinical and research settings. Med Sci Sports Exerc. 51 (7), 1538-1557 (2019).
  27. Meltzer, L. J., Montgomery-Downs, H. E., Insana, S. P., Walsh, C. M. Use of actigraphy for assessment in pediatric sleep research. Sleep Med Rev. 16 (5), 463-475 (2012).
  28. Paradiso, C., Colino, F., Liu, S. The validity and reliability of the mi band wearable device for measuring steps and heart rate. Int J Exerc Sci. 13 (4), 689-701 (2020).
  29. Benedetto, S. et al. Assessment of the fitbit charge 2 for monitoring heart rate. PloS One. 13 (2), e0192691 (2018).
  30. Chow, H. -W. Yang, C. -C. Accuracy of optical heart rate sensing technology in wearable fitness trackers for young and older adults: Validation and comparison study. JMIR Mhealth Uhealth. 8 (4), e14707 (2020).
  31. Jachymek, M. et al. Wristbands in home-based rehabilitation-validation of heart rate measurement. Sensors. 22 (1), 60 (2021).
  32. Schalk, M., Schalk, I., Bauernhansl, T., Siegert, J., Schneider, U. Investigation of possible effects of wearing exoskeletons during welding on heart rate. Physiologia. 2 (3), 94-108 (2022).
  33. Bent, B., Goldstein, B. A., Kibbe, W. A., Dunn, J. P. Investigating sources of inaccuracy in wearable optical heart rate sensors. NPJ Digit Med. 3 (1), 18 (2020).
  34. De Cocker, K. A., De Meyer, J., De Bourdeaudhuij, I. M., Cardon, G. M. Non-traditional wearing positions of pedometers: Validity and reliability of the Omron HJ-203-ED pedometer under controlled and free-living conditions. J Sci Med Sport. 15 (5), 418-424 (2012).
  35. Diaz, K. M. et al. Fitbit®: An accurate and reliable device for wireless physical activity tracking. Int J Cardiol. 185, 138-140 (2015).
  36. Mendes, M. Pala, A. Type I error rate and power of three normality tests. Inf Technol J. 2 135-139 (2003).
  37. Costa, R. R. et al. Heart rate deflection point as an alternative to determining the anaerobic threshold in dyslipidaemic patients. Motriz: Rev Educ Fís. 25 (1), (2019).
  38. Roth, S. et al. Noninvasive evaluation of the hemodynamic status in patients after heart transplantation or left ventricular assist device implantation. PloS One. 17 (10), e0275977 (2022).

Reprints and Permissions

Request permission to reuse the text or figures of this JoVE article

Request Permission

Explore More Articles

Fitness SmartwatchData AccuracyCardiovascular MonitoringPhysical Activity MonitoringValidation StudyHeart RateBlood Oxygen SaturationSleep DataPolysomnographyStep Count Validation3 Minute Walk TestStairs ClimbingWearable TechnologyClinical ContextData Reliability

This article has been published

Video Coming Soon

JoVE Logo

Privacy

Terms of Use

Policies

Contact Us

Recommend to library

JoVE NEWSLETTERS

Research

Education

Authors

Librarians

ABOUT JoVE

Copyright © 2025 MyJoVE Corporation. All rights reserved