Name: design and evaluation of smart glasses for food intake and physical activity classification
Uploaded: 2018-02-14T13:00:00
Description: Watch this Scientific Journal Video about Design and Evaluation of Smart Glasses for Food Intake and Physical Activity Classification at JoVE.com

This work was supported by Envisible, Inc. This study was also supported by a grant of the Korean Health Technology R&#38;D Project, Ministry of Health &#38; Welfare, Republic of Korea (HI15C1027). This research was also supported by the National Research Foundation of Korea (NRF-2016R1A1A1A05005348).

NOTE: All the procedures including the use of human subjects were accomplished by a non-invasive manner of simply wearing a pair of glasses. All the data were acquired by measuring the force signals from load cells inserted in the glasses that were not in direct contact with the skin. The data were wirelessly transmitted to the data recording module, which, in this case is a designated smartphone for the study. All the protocols were not related to in vivo/in vitro human studies. No drug and blood sampl...

<ol>
	<li>Sharma, A. M., Padwal, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Obesity+is+a+sign-over-eating+is+a+symptom:+an+aetiological+framework+for+the+assessment+and+management+of+obesity.">Obesity is a sign-over-eating is a symptom: an aetiological framework for the assessment and management of obesity.</a> Obes Rev. 11 (5), 362-370 (2010).</li><li>Pi-Sunyer, F. X., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Clinical+guidelines+on+the+identification,+evaluation,+and+treatment+of+overweight+and+obesity+in+adults.">Clinical guidelines on the identification, evaluation, and treatment of overweight and obesity in adults.</a> Am J Clin Nutr. 68 (4), 899-917 (1998).</li><li>McCrory, P., Strauss, B., Wahlqvist, M. L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Energy+balance,+food+intake+and+obesity.">Energy balance, food intake and obesity.</a> Exer Obes. , (1994).</li><li>Albinali, F., Intille, S., Haskell, W., Rosenberger, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Proceedings of the 12th ACM international conference on Ubiquitous computing. , 311-320 (2010).</li><li>Bonomi, A., Westerterp, K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Advances+in+physical+activity+monitoring+and+lifestyle+interventions+in+obesity:+a+review.">Advances in physical activity monitoring and lifestyle interventions in obesity: a review.</a> Int J Obes. 36 (2), 167-177 (2012).</li><li>Jung, S., Lee, J., Hyeon, T., Lee, M., Kim, D. H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Fabric-Based+Integrated+Energy+Devices+for+Wearable+Activity+Monitors.">Fabric-Based Integrated Energy Devices for Wearable Activity Monitors.</a> Adv Mater. 26 (36), 6329-6334 (2014).</li><li>Fulk, G. D., Sazonov, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Using+sensors+to+measure+activity+in+people+with+stroke.">Using sensors to measure activity in people with stroke.</a> Top Stroke Rehabil. 18 (6), 746-757 (2011).</li><li>Makeyev, O., Lopez-Meyer, P., Schuckers, S., Besio, W., Sazonov, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automatic+food+intake+detection+based+on+swallowing+sounds.">Automatic food intake detection based on swallowing sounds.</a> Biomed Signal Process Control. 7 (6), 649-656 (2012).</li><li>P&#228;&#223;ler, S., Fischer, W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Food+intake+activity+detection+using+an+artificial+neural+network.">Food intake activity detection using an artificial neural network.</a> Biomed Tech (Berl). , (2012).</li><li>Passler, S., Fischer, W. -. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Food+intake+monitoring:+Automated+chew+event+detection+in+chewing+sounds.">Food intake monitoring: Automated chew event detection in chewing sounds.</a> IEEE J Biomed Health Inform. 18 (1), 278-289 (2014).</li><li>Kadomura, A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> CHI'13 Extended Abstracts on Human Factors in Computing Systems. , 1551-1556 (2013).</li><li>Fontana, J. M., Farooq, M., Sazonov, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automatic+ingestion+monitor:+A+novel+wearable+device+for+monitoring+of+ingestive+behavior.">Automatic ingestion monitor: A novel wearable device for monitoring of ingestive behavior.</a> IEEE Trans Biomed Eng. 61 (6), 1772-1779 (2014).</li><li>Shen, Y., Salley, J., Muth, E., Hoover, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Assessing+the+Accuracy+of+a+Wrist+Motion+Tracking+Method+for+Counting+Bites+across+Demographic+and+Food+Variables.">Assessing the Accuracy of a Wrist Motion Tracking Method for Counting Bites across Demographic and Food Variables.</a> IEEE J Biomed Health Inform. , (2016).</li><li>Farooq, M., Sazonov, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> International Conference on Bioinformatics and Biomedical Engineering. , 464-472 (2017).</li><li>Grigoriadis, A., Johansson, R. S., Trulsson, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Temporal+profile+and+amplitude+of+human+masseter+muscle+activity+is+adapted+to+food+properties+during+individual+chewing+cycles.">Temporal profile and amplitude of human masseter muscle activity is adapted to food properties during individual chewing cycles.</a> J Oral Rehab. 41 (5), 367-373 (2014).</li><li>Strini, P. J. S. A., Strini, P. J. S. A., de Souza Barbosa, T., Gavi&#227;o, M. B. D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Assessment+of+thickness+and+function+of+masticatory+and+cervical+muscles+in+adults+with+and+without+temporomandibular+disorders.">Assessment of thickness and function of masticatory and cervical muscles in adults with and without temporomandibular disorders.</a> Arch Oral Biol. 58 (9), 1100-1108 (2013).</li><li>Standring, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Gray's anatomy: the anatomical basis of clinical practice. , (2015).</li><li>Farooq, M., Sazonov, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+novel+wearable+device+for+food+intake+and+physical+activity+recognition.">A novel wearable device for food intake and physical activity recognition.</a> Sensors. 16 (7), 1067 (2016).</li><li>Zhang, R., Amft, O. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Proceedings of the 2016 ACM International Symposium on Wearable Computers. , 50-52 (2016).</li><li>Farooq, M., Sazonov, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Segmentation+and+Characterization+of+Chewing+Bouts+by+Monitoring+Temporalis+Muscle+Using+Smart+Glasses+with+Piezoelectric+Sensor.">Segmentation and Characterization of Chewing Bouts by Monitoring Temporalis Muscle Using Smart Glasses with Piezoelectric Sensor.</a> IEEE J Biomed Health Inform. , (2016).</li><li>Huang, Q., Wang, W., Zhang, Q. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Your+Glasses+Know+Your+Diet:+Dietary+Monitoring+using+Electromyography+Sensors.">Your Glasses Know Your Diet: Dietary Monitoring using Electromyography Sensors.</a> IEEE Internet of Things Journal. , (2017).</li><li>Chung, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+glasses-type+wearable+device+for+monitoring+the+patterns+of+food+intake+and+facial+activity.">A glasses-type wearable device for monitoring the patterns of food intake and facial activity.</a> Scientific Reports. 7, 41690 (2017).</li><li>Cristianini, N., Shawe-Taylor, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> An introduction to support Vector Machines: and other kernel-based learning methods. , (2000).</li><li>Giannakopoulos, T., Pikrakis, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Introduction to Audio Analysis: A MATLAB Approach. , (2014).</li><li>Chang, C. -. C., Lin, C. -. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=LIBSVM:+a+library+for+support+vector+machines.">LIBSVM: a library for support vector machines.</a> ACM Trans Intell Syst Technol. 2 (3), 27 (2011).</li><li>Po, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Time-frequency+analysis+of+chewing+activity+in+the+natural+environment.">Time-frequency analysis of chewing activity in the natural environment.</a> J Dent Res. 90 (10), 1206-1210 (2011).</li><li>Ji, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Frequency+and+velocity+of+people+walking.">Frequency and velocity of people walking.</a> Struct Eng. 84 (3), 36-40 (2005).</li><li>Knoblauch, R., Pietrucha, M., Nitzburg, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Field+studies+of+pedestrian+walking+speed+and+start-up+time.">Field studies of pedestrian walking speed and start-up time.</a> Transp Res Red. 1538, 27-38 (1996).</li></ol>

In this study, we first proposed the design and manufacturing process of glasses that sense the patterns of food intake and physical activities. As this study mainly focused on the data analysis to distinguish the food intake from the other physical activities (such as walking and winking), the sensor and data acquisition system required the implementation of mobility recording. Thus, the system included the sensors, the MCU with wireless communication capability, and the battery. The proposed protocol provided a novel a...

Continuous and objective monitoring of food intake is essential for maintaining energy balance in the human body, as excessive energy accumulation may cause overweightness and obesity1, which could result in various medical complications2. The main factors in the energy imbalance are known to be both excessive food intake and insufficient physical activity3. Various studies on the monitoring of daily energy expenditure have been introduced with automatic and objective measurement of physical activity patterns through wearable devices4,5,6, even at the end-consumer level and medical stage7. Research on the monitoring of food intake, however, is still in the laboratory setting, since it is difficult to detect the food intake activity in a direct and objective manner. Here, we aim to present a device design and its evaluation for monitoring the food intake and physical activity patterns at a practical level in daily life.
There have been various indirect approaches to monitor the food intake through chewing and swallowing sounds8,9,10, movement of the wrist11,12,13, image analysis14, and electromyogram (EMG)15. However, these approaches were difficult to apply to daily life applications, because of their inherent limitations: the methods using sound were vulnerable to be influenced by environmental sound; the methods using the movement of the wrist were difficult to distinguish from other physical activities when not consuming food; and the methods using the images and EMG signals are restricted by the boundary of movement and environment. These studies showed the capability of automated detection of the food intake using sensors, but still had a limitation of practical applicability to everyday life beyond laboratory settings.
In this study, we used the patterns of temporalis muscle activity as the automatic and objective monitoring of the food intake. In general, the temporalis muscle repeats the contraction and relaxation as a part of masticatory muscle during the food intake16,17; thus, the food intake activity can be monitored by detecting the periodic patterns of temporalis muscle activity. Recently, there have been several studies utilizing the temporalis muscle activity18,19,20,21, which used the EMG or piezoelectric strain sensor and attaching them directly onto human skin. These approaches, however, were sensitive to the skin location of the EMG electrodes or strain sensors, and were easily detached from the skin due to the physical movement or perspiration. Therefore, we proposed a new and effective method using a pair of glasses that sense the temporalis muscle activity through two load cells inserted in both the hinges in our previous study22. This method showed great potential of detecting the food intake activity with a high accuracy without touching the skin. It was also un-obtrusive and non-intrusive, since we used a common glasses-type device.
In this study, we present a series of detailed protocols of how to implement the glasses-type device and how to use the patterns of temporalis muscle activity for monitoring the food intake and physical activity. The protocols include the process of hardware design and fabrication that consists of a 3D-printed frame of the glasses, a circuit module, and a data acquisition module, and include the software algorithms for data processing and analysis. We furthermore examined the classification among several featured activities (e.g., chewing, walking, and winking) to demonstrate the potential as a practical system that can tell a minute difference between the food intake and other physical activity patterns.

<table>
	<tbody>
		<tr>
			<td>FSS1500NSB</td>
			<td>Honeywell, USA</td>
			<td></td>
			<td>Load cell</td>
		</tr>
		<tr>
			<td>INA125U</td>
			<td>Texas Instruments, USA</td>
			<td></td>
			<td>Amplifier</td>
		</tr>
		<tr>
			<td>ESP-07</td>
			<td>Shenzhen Anxinke Technology, China</td>
			<td></td>
			<td>MCU with Wi-Fi module</td>
		</tr>
		<tr>
			<td>74LVC1G3157</td>
			<td>Nexperia, The Netherlands</td>
			<td></td>
			<td>Multiplexer</td>
		</tr>
		<tr>
			<td>MP701435P</td>
			<td>Maxpower, China</td>
			<td></td>
			<td>LiPo battery</td>
		</tr>
		<tr>
			<td>U1V10F3</td>
			<td>Pololu, USA</td>
			<td></td>
			<td>Voltage regulator</td>
		</tr>
		<tr>
			<td>Ultimaker 2</td>
			<td>Ultimaker, The Netherlands</td>
			<td></td>
			<td>3D printer</td>
		</tr>
		<tr>
			<td>ColorFabb XT-CF20</td>
			<td>ColorFabb, The Netherlands</td>
			<td></td>
			<td>Carbon fiber filament</td>
		</tr>
		<tr>
			<td>iPhone 6s Plus</td>
			<td>Apple, USA</td>
			<td></td>
			<td>Data acquisition device</td>
		</tr>
		<tr>
			<td>Jelly Belly</td>
			<td>Jelly Belly Candy Company, USA</td>
			<td></td>
			<td>Food texture for user study</td>
		</tr>
	</tbody>
</table>

design and evaluation of smart glasses for food intake and physical activity classification

This study presents a series of protocols of designing and manufacturing a glasses-type wearable device that detects the patterns of temporalis muscle activities during food intake and other physical activities. We fabricated a 3D-printed frame of the glasses and a load cell-integrated printed circuit board (PCB) module inserted in both hinges of the frame. The module was used to acquire the force signals, and transmit them wirelessly. These procedures provide the system with higher mobility, which can be evaluated in practical wearing conditions such as walking and waggling. A performance of the classification is also evaluated by distinguishing the patterns of food intake from those physical activities. A series of algorithms were used to preprocess the signals, generate feature vectors, and recognize the patterns of several featured activities (chewing and winking), and other physical activities (sedentary rest, talking, and walking). The results showed that the average F1 score of the classification among the featured activities was 91.4%. We believe this approach can be potentially useful for automatic and objective monitoring of ingestive behaviors with higher accuracy as practical means to treat ingestive problems.

Through the procedures outlined in the protocol, we prepared two versions of the 3D printed frame by differentiating the length of the head piece, LH (133 and 138 mm), and the temples, LT (110 and 125 mm), as shown in Figure 4. Therefore, we can cover several wearing conditions, which can be varied from the subjects&#39; head size, shape, etc. The subjects chose one of the frames to fit to their head for the user study. The vert...

Watch this Scientific Journal Video about Design and Evaluation of Smart Glasses for Food Intake and Physical Activity Classification at JoVE.com

Design and Evaluation of Smart Glasses for Food Intake and Physical Activity Classification

This study presents a protocol of designing and manufacturing a glasses-type wearable device that detects the patterns of food intake and other featured physical activities using load cells inserted in both hinges of the glasses.

This study presents a series of protocols of designing and manufacturing a glasses-type wearable device that detects the patterns of temporalis muscle ...

Research

JoVE Journal

Engineering

Experiments on Ultrasonic Lubrication Using a Piezoelectrically-assisted Tribometer and Optical Profilometer

Friction and wear are detrimental to engineered systems. Ultrasonic lubrication is achieved when the interface between two sliding surfaces is vibrated at ...

Design and Use of a Full Flow Sampling System (FFS) for the Quantification of Methane Emissions

Design and Use of a Full Flow Sampling System FFS for the Quantification of Methane Emissions

The use of natural gas continues to grow with increased discovery and production of unconventional shale resources. At the same time, the natural gas ...

Scanning Light Scattering Profiler (SLPS) Based Methodology to Quantitatively Evaluate Forward and Backward Light Scattering from Intraocular Lenses

Scanning Light Scattering Profiler SLPS Based Methodology to Quantitatively Evaluate Forward and Backward Light Scattering from Intraocular Lenses

The scanning light scattering profiler (SLSP) methodology has been developed for the full-angle quantitative evaluation of forward and backward light ...

Measuring the Densities of Aqueous Glasses at Cryogenic Temperatures

We demonstrate a method for determining the vitreous phase cryogenic temperature densities of aqueous mixtures, and other samples that require rapid ...

Design and Characterization Methodology for Efficient Wide Range Tunable MEMS Filters

Here, we demonstrate the advantages of the laser Doppler vibrometer (LDV) over conventional techniques (the network analyzer), as well as the techniques ...

Methods of Ex Situ and In Situ Investigations of Structural Transformations: The Case of Crystallization of Metallic Glasses

Methods of Ex Situ and In Situ Investigations of Structural Transformations: The Case of Crystallization of Metallic Glasses

We demonstrate the use of two nuclear-based analytical methods that can follow the modifications of microstructural arrangement of iron-based metallic ...

Hybrid Printing for the Fabrication of Smart Sensors

A method to combine additively manufactured substrates or foils and multilayer inkjet printing for the fabrication of sensor devices is presented. First, ...

Design and Implementation of a Bespoke Robotic Manipulator for Extra-corporeal Ultrasound

With the potential for high precision, dexterity, and repeatability, a self-tracked robotic system can be employed to assist the acquisition of real-time ...

Crack Monitoring in Resonance Fatigue Testing of Welded Specimens Using Digital Image Correlation

A procedure using digital image correlation (DIC) to detect cracks on welded specimens during fatigue tests on resonance testing machines is presented. It ...