Name: design and evaluation of smart glasses for food intake and physical activity classification
Uploaded: 2018-02-14T13:00:00.000+00:00
Duration: 7 min 47 s
Description: Watch this Scientific Journal Video about Design and Evaluation of Smart Glasses for Food Intake and Physical Activity Classification at JoVE.com

This work was supported by Envisible, Inc. This study was also supported by a grant of the Korean Health Technology R&#38;D Project, Ministry of Health &#38; Welfare, Republic of Korea (HI15C1027). This research was also supported by the National Research Foundation of Korea (NRF-2016R1A1A1A05005348).

NOTE: All the procedures including the use of human subjects were accomplished by a non-invasive manner of simply wearing a pair of glasses. All the data were acquired by measuring the force signals from load cells inserted in the glasses that were not in direct contact with the skin. The data were wirelessly transmitted to the data recording module, which, in this case is a designated smartphone for the study. All the protocols were not related to in vivo/in vitro human studies. No drug and blood samples were used for the experiments. Informed consent was obtained from all subjects of the experiments.
1. Manufacturing of a Sensor-integrated Circuit Module
<ol>
	<li>Purchase electronic components for manufacturing the circuit module.
	<ol>
		<li>Purchase two ball-type load cells, each of which operates in a range between 0 N and 15 N, and produces an output of low differential voltage with maximum 120 mV span in a 3.3 V excitation. 
		NOTE: These load cells are used to measure force signals on both the left and right sides of the glasses.</li>
		<li>Purchase two instrumentation amplifiers and two 15 k&#937; gain-setting resistors. 
		NOTE: The instrumentation amplifier and the gain-setting resistor are used to amplify the force signal of the load cell eight times, up to 960 mV.</li>
		<li>Purchase a micro controller unit (MCU) with wireless capability (e.g., Wi-Fi connectivity), and a 10-bit analog-to-digital converter (ADC). 
		NOTE: The MCU is used to read the force signals and transmit them to a data acquisition module wirelessly. Because one analog input pin is used for two analog force inputs, the use of a multiplexer is introduced in the next step 1.1.4.</li>
		<li>Purchase a two-channel analog multiplexer that handles the two input signals with one ADC pin on the MCU.</li>
		<li>Purchase a lithium-ion polymer (LiPo) battery with 3.7 V nominal voltage, 300 mAh nominal capacity, and 1 C discharge rate. 
		NOTE: The battery capacity was chosen to supply enough current per hour more than 200 mAh and to operate the system reliably for about 1.5 h of an experiment.</li>
		<li>Purchase a 3.3 V voltage regulator for linear down-regulation of the 3.7 V battery voltage to the 3.3 V operating voltage of the system.</li>
		<li>Purchase five 12 k&#937; surface-mounted devices (SMD) type resistors as pull-up resistors of the MCU. The resistor&#39;s footprint is 2.0 mm x 1.2 mm (size 2012).</li>
	</ol>
	</li>
	<li>Fabricate printed circuit boards (PCBs). This step is about drawing the circuit boards, and making the artwork (i.e., the board layout, the .brd file) and the schematic (i.e., the .sch file) for PCB fabrication. A basic understanding of the process of creating artwork and schematic files is required for development.
	<ol>
		<li>Draw a schematic of a left circuit containing the battery using an electronic design application as shown in Figure 1A. Save the result as both artwork (.brd) and schematic (.sch) files.</li>
		<li>Draw a schematic of a right circuit containing the MCU using an electronic design application as shown in Figure 1B. Save the result as both artwork (.brd) and schematic (.sch) files.</li>
		<li>Fabricate the circuit boards by placing an order with a PCB fabrication company.</li>
		<li>Solder every electronic component prepared in step 1.1 to the PCBs as shown in Figure 2 and Figure 3. 
		CAUTION: The instrumentation amplifier is very sensitive to the soldering temperature. Make sure that lead temperature does not exceed 300 &#176;C for 10 s during soldering, otherwise it may cause permanent damage to the component.</li>
	</ol>
	</li>
</ol>
2. 3D Printing of a Frame of the Glasses
<ol>
	<li>Draw the 3D model of the head piece of the glasses using a 3D modeling tool as shown in Figure 4A. Export the result to the .stl file format.</li>
	<li>Draw the 3D model of the left and right temples of the glasses using a 3D modeling tool as shown in Figure 4B and Figure 4C. Export the results to the .stl file format.</li>
	<li>Print the head piece and temple parts using a 3D printer and a carbon fiber filament at 240 &#176;C of a nozzle temperature and 80 &#176;C of a bed temperature. 
	NOTE: The use of any commercial 3D printer and any types of filaments such as acrylonitrile butadiene styrene (ABS) and polylactide (PLA) can be permitted. The nozzle and bed temperatures may be adjusted according to the filament and printing conditions.</li>
	<li>Heat the tips of the temples using a hot air blower of a 180 &#176;C setting and bend them inward about 15 degrees to contact the epidermis of the temporalis muscle like conventional glasses. 
	NOTE: The degree of bending of the glasses temple does not need to be rigorous as the purpose of the curvature is to increase a form factor by helping the glasses fit on a subject&#39;s head when equipped. Be careful, however, as excessive bending will prevent the temples from touching the temporalis muscle, which makes it impossible to collect significant patterns.</li>
	<li>Repeat the steps from step 2.1&#8211;2.4 to print two different sizes of the glasses frame to fit multiple head sizes as shown in Figure 4.</li>
</ol>
3. Assembly of All Parts of the Glasses
<ol>
	<li>Insert the PCBs on both sides of the temples of the glasses using M2 bolts as shown in Figure 5.</li>
	<li>Assemble the head piece and the temples by inserting the M2 bolts into the hinge joints.</li>
	<li>Connect the left and right PCBs using the 3-pin connecting wires as shown in Figure 5.</li>
	<li>Connect the battery to the left circuit and attach it with an adhesive tape to the left temple. The mounting side of the battery is not critical, since it may vary depending on the PCB design.</li>
	<li>Cover the glasses with rubber tapes on the tip and the nose pad to add more friction with the human skin as shown in Figure 5.</li>
</ol>
4. Development of a Data Acquisition System
NOTE: The data acquisition system is composed of a data transmitting module and a data receiving module. The data transmitting module reads the time and the force signals of both sides, and then sends them to the data receiving module, which gathers the received data and writes them to .tsv files.
<ol>
	<li>Upload the data transmitting application to the MCU of the PCB module following the procedures in steps 4.1.1&#8211;4.1.3.
	<ol>
		<li>Run the &#34;GlasSense_Server&#34; project attached to the supplementary files using a computer. 
		NOTE: This project was built with Arduino integrated development environment (IDE). It provides the ability to read the time and force signals with 200 samples/s, and transmit them to the data receiving module.</li>
		<li>Connect the PCB module to the computer via a universal serial bus (USB) connector.</li>
		<li>Press the &#34;Upload&#34; button on the Arduino IDE to flash the programming codes from step 4.1.1 into the MCU.</li>
	</ol>
	</li>
	<li>Upload the data receiving application to a smartphone, which is used to receive the data wirelessly, following the procedures in steps 4.2.1&#8211;4.2.3.
	<ol>
		<li>Run the &#34;GlasSense_Client&#34; project attached to the supplementary files using a computer. 
		NOTE: This project was built with C# programming language. It provides the ability to receive data and save the .tsv files, which contain a subject&#39;s information, such as name, sex, age, and body mass index (BMI).</li>
		<li>Connect the smartphone to the computer via a USB connector to build the data receiving application.</li>
		<li>Press the &#34;File &#62; Build &#38; Run&#34; button on the C# project to build the data receiving application to the smartphone.</li>
	</ol>
	</li>
</ol>
5. Data Collection from a User Study
NOTE: This study collected six featured activity sets: sedentary rest (SR), sedentary chewing (SC), walking (W), chewing while walking (CW), sedentary talking (ST), and sedentary wink (SW).
<ol>
	<li>Select a pair of glasses which have an appropriate size to the user to be tested. Fine-tune the tightness with the support bolts at both the hinges (Figure 5). 
	CAUTION: The force values must not exceed 15 N, since the force sensors used in this study may lose the fine linear characteristic beyond the operation range. The force values can be fine-tuned by loosening or tightening the support bolts.</li>
	<li>Record the activities of all subjects by pressing the &#34;Record&#34; button on the application built in step 4.2.3.
	<ol>
		<li>Record an activity during a 120-s block and generate a recording file of it.
		<ol>
			<li>In the case of SR, sit the subject in a chair and have them use a smartphone or read a book. Allow movement of the head, but avoid movement of the whole body.</li>
			<li>In the cases of SC and CW, have the subjects eat two types of food texture (toasted bread and chewing jelly) in order to reflect different food properties. Serve the toasted bread in slices of 20 mm x 20 mm, which is a good size for eating.</li>
			<li>In the case of W, have the subjects walk at a speed of 4.5 km/h on a treadmill.</li>
			<li>In the case of ST, sit the subjects down and have them read a book out loud in a normal tone and speed.</li>
			<li>In the case of SW, inform the subjects to wink on the timing of a bell sound of 0.5 s long every 3 s.</li>
		</ol>
		</li>
		<li>Generate a recording file in .tsv format from the data collected in step 5.2.1. 
		NOTE: This file contains a sequence of the time when the data were received, a left force signal, a right force signal, and a label representing the current facial activity. Visualizations of temporal signals of all activities in a block of a user were depicted in Figure 6. The six featured activity sets (SR, SC, W, CW, ST, and SW) were labeled as 1, 2, 3, 4, 5, and 6, respectively. The labels were used to compare the predicted classes in section 8 of the protocol.</li>
		<li>Take a 60-s break after the recording block. Take off the glasses during the break, and re-wear them again when the recording block restarts.</li>
		<li>Repeat the block-and-break set of steps 5.2.1 and 5.2.2 four times for each activity.</li>
		<li>In the case of SW, have the subject wink repeatedly with the left eye during one block, and then wink repeatedly with the right eye during the next block.</li>
	</ol>
	</li>
	<li>Repeat steps 5.1&#8211;5.2 and collect the data from 10 subjects. In this study, we used five males and five females, the average age was 27.9 &#177; 4.3 (standard deviation; s.d.) years, which ranged at 19&#8211;33 years, and the average BMI was 21.6 &#177; 3.2 (s.d.) kg/m2, which ranged at 17.9&#8211;27.4 kg/m2. 
	NOTE: In this study, the subjects who did not have any medical conditions to chew food, wink, and walk were recruited, and this condition was used for inclusion criteria.</li>
</ol>
6. Signal Preprocessing and Segmentation
NOTE: The left and right signals are calculated separately in the following procedures.
<ol>
	<li>Prepare a series of temporal frames of 2 s long.
	<ol>
		<li>Segment the 120 s recorded signals into a set of 2 s frames by hopping them at 1-s intervals using MATLAB as shown in Figure 6. 
		NOTE: The segmented frames of 2 s long were used to extract features in section 7. The 1 s hopping size was determined to divide the signals by the 3 s wink interval already mentioned in step 5.2.1.</li>
		<li>Apply a low-pass filter (LPF) using a 5th order Butterworth filter with a cutoff frequency of 10 Hz for each frame.</li>
		<li>Save the results of step 6.1.2 as the temporal frames for the next steps in step 7.1.</li>
	</ol>
	</li>
	<li>Prepare a series of spectral frames.
	<ol>
		<li>Subtract the median from the original signals of each frame to remove the preload when wearing the glasses. 
		NOTE: The preload value is not required for the following frequency analysis, since it does not include any information about chewing, walking, wink, etc. It could, however, contain significant information, which can vary from subject to subject, from every setting of the glasses, and even from the moment a subject wears the glasses.</li>
		<li>Apply a Hanning window to each frame to reduce a spectral leakage on frequency analysis.</li>
		<li>Produce and save a single-sided spectrum by applying a fast Fourier transform (FFT) to each frame.</li>
	</ol>
	</li>
	<li>Define a combination of a temporal and a spectral frame of the same time as a frame block (or simply a frame).</li>
</ol>
7. Generation of Feature Vectors
NOTE: A feature vector is generated per frame produced in section 6 of the protocol. The left and right frames are calculated separately and combined into a feature vector in the following procedures. All the procedures were implemented in MATLAB.
<ol>
	<li>Extract statistical features from a temporal frame in step 6.1 of the protocol. A list of the total number of 54 features is given in Table 1.</li>
	<li>Extract statistical features from a spectral frame in step 6.2 of the protocol. A list of the total number of 30 features is given in Table 2.</li>
	<li>Generate an 84-dimensional feature vector by combining the temporal and spectral features above.</li>
	<li>Label the generated feature vectors from the recordings in step 5.2 of the protocol.</li>
	<li>Repeat the steps from steps 7.1&#8211;7.4 for all frame blocks and generate a series of feature vectors.</li>
</ol>
8. Classification of the Activities into Classes
NOTE: This step is to select the classifier model of a support vector machine (SVM)23 by determining parameters that show the best accuracy from the given problem (i.e., feature vectors). The SVM is a well-known supervised machine learning technique, which shows excellent performance in generalization and robustness using a maximum margin between the classes and a kernel function. We used a grid-search and a cross-validation method to define a penalty parameter C and a kernel parameter &#947; of the radial basis function (RBF) kernel. A minimum understanding of machine learning techniques and the SVM is required to perform the following procedures. Some referential materials23,24,25 are recommended for better understanding of machine learning techniques and the SVM algorithm. All the procedures in this section were implemented using LibSVM25 software package.
<ol>
	<li>Define a grid of pairs of (C, &#947;) for the grid-search. Use exponentially growing sequences of C (2-10, 2-5, &#8230;, 230) and &#947; (2-30, 2-25, &#8230;, 210). 
	NOTE: These sequences were determined heuristically.</li>
	<li>Define a pair of (C, &#947;) (e.g., (2-10, 2-30)).</li>
	<li>For the defined grid in step 8.2, perform the 10-fold cross-validation scheme. 
	NOTE: This scheme divides the entire feature vectors into 10-part subsets, then test one subset from the classifier model trained by the other subsets, and repeat it over all the subsets, one by one. Therefore, every feature vectors can be tested sequentially.
	<ol>
		<li>Divide the entire feature vectors into 10-part subsets.</li>
		<li>Define a testing set from a subset, and a training set from the remaining 9 subsets.</li>
		<li>Define a scale vector that scales all elements of the feature vectors to the range of [0, 1] for the training set. 
		NOTE: The scale vector has same dimension with the feature vector. It consists of a set of multipliers that scales the same row (or column) of all feature vectors to the range of [0, 1]. For example, the first feature of a feature vector is linearly scaled to the range of [0, 1] for the all first features of the training feature vectors. Note that the scale vector is defined from the training set, because the testing set should be assumed to be unknown. This step increases the accuracy of the classification by making the features the equal range and avoiding numerical errors during the calculation.</li>
		<li>Scale each feature of the training set to the range of [0, 1] using the scale vector obtained in step 8.2.3.</li>
		<li>Scale each feature of the testing set to the range of [0, 1] using the scale vector obtained in step 8.2.3.</li>
		<li>Train the training set through the SVM with the defined pair of (C, &#947;) in step 8.2, and then build a classifier model.</li>
		<li>Test the testing set through the SVM with the defined pair of (C, &#947;) in step 8.2, and the classifier model obtained from the training procedure.</li>
		<li>Calculate a classification accuracy on the testing set. The accuracy was calculated from the percentage of feature vectors which are correctly classified.</li>
		<li>Repeat the steps 8.2.2&#8211;8.2.8 for all the subsets, and calculate the average accuracy of all subsets.</li>
	</ol>
	</li>
	<li>Repeat the steps 8.2&#8211;8.3.9 for all grid points of a pair of (C, &#947;).</li>
	<li>Find the local maximum of the highest accuracy of the grid. All the procedures of section 8 are illustrated in Figure 7.</li>
	<li>(Optional) If the step of the grid is considered coarse, repeat the steps 8.1&#8211;8.5 in a finer grid near the local maximum found in step 8.5, and find a new local maximum of the fine grid.</li>
	<li>Compute the precision, recall, and F1 score of each class of activities from the following equations: 
	<img alt="Equation 1" src="/files/ftp_upload/56633/56633eq1.jpg" />&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;Equation 1 
	<img alt="Equation 2" src="/files/ftp_upload/56633/56633eq2.jpg" />&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;Equation 2 
	<img alt="Equation 3" src="/files/ftp_upload/56633/56633eq3.jpg" />&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;Equation 3 
	where TP, FP, and FN represent true positives, false positives, and false negatives for each activity, respectively. The confusion matrix of all the activities is given in Table 3.</li>
</ol>

<ol>
	<li>Sharma, A. M., Padwal, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Obesity+is+a+sign-over-eating+is+a+symptom:+an+aetiological+framework+for+the+assessment+and+management+of+obesity.">Obesity is a sign-over-eating is a symptom: an aetiological framework for the assessment and management of obesity.</a> Obes Rev. 11 (5), 362-370 (2010).</li><li>Pi-Sunyer, F. X., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Clinical+guidelines+on+the+identification,+evaluation,+and+treatment+of+overweight+and+obesity+in+adults.">Clinical guidelines on the identification, evaluation, and treatment of overweight and obesity in adults.</a> Am J Clin Nutr. 68 (4), 899-917 (1998).</li><li>McCrory, P., Strauss, B., Wahlqvist, M. L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Energy+balance,+food+intake+and+obesity.">Energy balance, food intake and obesity.</a> Exer Obes. , (1994).</li><li>Albinali, F., Intille, S., Haskell, W., Rosenberger, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Proceedings of the 12th ACM international conference on Ubiquitous computing. , 311-320 (2010).</li><li>Bonomi, A., Westerterp, K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Advances+in+physical+activity+monitoring+and+lifestyle+interventions+in+obesity:+a+review.">Advances in physical activity monitoring and lifestyle interventions in obesity: a review.</a> Int J Obes. 36 (2), 167-177 (2012).</li><li>Jung, S., Lee, J., Hyeon, T., Lee, M., Kim, D. H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Fabric-Based+Integrated+Energy+Devices+for+Wearable+Activity+Monitors.">Fabric-Based Integrated Energy Devices for Wearable Activity Monitors.</a> Adv Mater. 26 (36), 6329-6334 (2014).</li><li>Fulk, G. D., Sazonov, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Using+sensors+to+measure+activity+in+people+with+stroke.">Using sensors to measure activity in people with stroke.</a> Top Stroke Rehabil. 18 (6), 746-757 (2011).</li><li>Makeyev, O., Lopez-Meyer, P., Schuckers, S., Besio, W., Sazonov, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automatic+food+intake+detection+based+on+swallowing+sounds.">Automatic food intake detection based on swallowing sounds.</a> Biomed Signal Process Control. 7 (6), 649-656 (2012).</li><li>P&#228;&#223;ler, S., Fischer, W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Food+intake+activity+detection+using+an+artificial+neural+network.">Food intake activity detection using an artificial neural network.</a> Biomed Tech (Berl). , (2012).</li><li>Passler, S., Fischer, W. -. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Food+intake+monitoring:+Automated+chew+event+detection+in+chewing+sounds.">Food intake monitoring: Automated chew event detection in chewing sounds.</a> IEEE J Biomed Health Inform. 18 (1), 278-289 (2014).</li><li>Kadomura, A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> CHI'13 Extended Abstracts on Human Factors in Computing Systems. , 1551-1556 (2013).</li><li>Fontana, J. M., Farooq, M., Sazonov, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automatic+ingestion+monitor:+A+novel+wearable+device+for+monitoring+of+ingestive+behavior.">Automatic ingestion monitor: A novel wearable device for monitoring of ingestive behavior.</a> IEEE Trans Biomed Eng. 61 (6), 1772-1779 (2014).</li><li>Shen, Y., Salley, J., Muth, E., Hoover, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Assessing+the+Accuracy+of+a+Wrist+Motion+Tracking+Method+for+Counting+Bites+across+Demographic+and+Food+Variables.">Assessing the Accuracy of a Wrist Motion Tracking Method for Counting Bites across Demographic and Food Variables.</a> IEEE J Biomed Health Inform. , (2016).</li><li>Farooq, M., Sazonov, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> International Conference on Bioinformatics and Biomedical Engineering. , 464-472 (2017).</li><li>Grigoriadis, A., Johansson, R. S., Trulsson, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Temporal+profile+and+amplitude+of+human+masseter+muscle+activity+is+adapted+to+food+properties+during+individual+chewing+cycles.">Temporal profile and amplitude of human masseter muscle activity is adapted to food properties during individual chewing cycles.</a> J Oral Rehab. 41 (5), 367-373 (2014).</li><li>Strini, P. J. S. A., Strini, P. J. S. A., de Souza Barbosa, T., Gavi&#227;o, M. B. D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Assessment+of+thickness+and+function+of+masticatory+and+cervical+muscles+in+adults+with+and+without+temporomandibular+disorders.">Assessment of thickness and function of masticatory and cervical muscles in adults with and without temporomandibular disorders.</a> Arch Oral Biol. 58 (9), 1100-1108 (2013).</li><li>Standring, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Gray's anatomy: the anatomical basis of clinical practice. , (2015).</li><li>Farooq, M., Sazonov, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+novel+wearable+device+for+food+intake+and+physical+activity+recognition.">A novel wearable device for food intake and physical activity recognition.</a> Sensors. 16 (7), 1067 (2016).</li><li>Zhang, R., Amft, O. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Proceedings of the 2016 ACM International Symposium on Wearable Computers. , 50-52 (2016).</li><li>Farooq, M., Sazonov, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Segmentation+and+Characterization+of+Chewing+Bouts+by+Monitoring+Temporalis+Muscle+Using+Smart+Glasses+with+Piezoelectric+Sensor.">Segmentation and Characterization of Chewing Bouts by Monitoring Temporalis Muscle Using Smart Glasses with Piezoelectric Sensor.</a> IEEE J Biomed Health Inform. , (2016).</li><li>Huang, Q., Wang, W., Zhang, Q. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Your+Glasses+Know+Your+Diet:+Dietary+Monitoring+using+Electromyography+Sensors.">Your Glasses Know Your Diet: Dietary Monitoring using Electromyography Sensors.</a> IEEE Internet of Things Journal. , (2017).</li><li>Chung, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+glasses-type+wearable+device+for+monitoring+the+patterns+of+food+intake+and+facial+activity.">A glasses-type wearable device for monitoring the patterns of food intake and facial activity.</a> Scientific Reports. 7, 41690 (2017).</li><li>Cristianini, N., Shawe-Taylor, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> An introduction to support Vector Machines: and other kernel-based learning methods. , (2000).</li><li>Giannakopoulos, T., Pikrakis, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Introduction to Audio Analysis: A MATLAB Approach. , (2014).</li><li>Chang, C. -. C., Lin, C. -. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=LIBSVM:+a+library+for+support+vector+machines.">LIBSVM: a library for support vector machines.</a> ACM Trans Intell Syst Technol. 2 (3), 27 (2011).</li><li>Po, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Time-frequency+analysis+of+chewing+activity+in+the+natural+environment.">Time-frequency analysis of chewing activity in the natural environment.</a> J Dent Res. 90 (10), 1206-1210 (2011).</li><li>Ji, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Frequency+and+velocity+of+people+walking.">Frequency and velocity of people walking.</a> Struct Eng. 84 (3), 36-40 (2005).</li><li>Knoblauch, R., Pietrucha, M., Nitzburg, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Field+studies+of+pedestrian+walking+speed+and+start-up+time.">Field studies of pedestrian walking speed and start-up time.</a> Transp Res Red. 1538, 27-38 (1996).</li></ol>

The authors have nothing to disclose.

In this study, we first proposed the design and manufacturing process of glasses that sense the patterns of food intake and physical activities. As this study mainly focused on the data analysis to distinguish the food intake from the other physical activities (such as walking and winking), the sensor and data acquisition system required the implementation of mobility recording. Thus, the system included the sensors, the MCU with wireless communication capability, and the battery. The proposed protocol provided a novel a...

Continuous and objective monitoring of food intake is essential for maintaining energy balance in the human body, as excessive energy accumulation may cause overweightness and obesity1, which could result in various medical complications2. The main factors in the energy imbalance are known to be both excessive food intake and insufficient physical activity3. Various studies on the monitoring of daily energy expenditure have been introduced with automatic and objective measurement of physical activity patterns through wearable devices4,5,6, even at the end-consumer level and medical stage7. Research on the monitoring of food intake, however, is still in the laboratory setting, since it is difficult to detect the food intake activity in a direct and objective manner. Here, we aim to present a device design and its evaluation for monitoring the food intake and physical activity patterns at a practical level in daily life.
There have been various indirect approaches to monitor the food intake through chewing and swallowing sounds8,9,10, movement of the wrist11,12,13, image analysis14, and electromyogram (EMG)15. However, these approaches were difficult to apply to daily life applications, because of their inherent limitations: the methods using sound were vulnerable to be influenced by environmental sound; the methods using the movement of the wrist were difficult to distinguish from other physical activities when not consuming food; and the methods using the images and EMG signals are restricted by the boundary of movement and environment. These studies showed the capability of automated detection of the food intake using sensors, but still had a limitation of practical applicability to everyday life beyond laboratory settings.
In this study, we used the patterns of temporalis muscle activity as the automatic and objective monitoring of the food intake. In general, the temporalis muscle repeats the contraction and relaxation as a part of masticatory muscle during the food intake16,17; thus, the food intake activity can be monitored by detecting the periodic patterns of temporalis muscle activity. Recently, there have been several studies utilizing the temporalis muscle activity18,19,20,21, which used the EMG or piezoelectric strain sensor and attaching them directly onto human skin. These approaches, however, were sensitive to the skin location of the EMG electrodes or strain sensors, and were easily detached from the skin due to the physical movement or perspiration. Therefore, we proposed a new and effective method using a pair of glasses that sense the temporalis muscle activity through two load cells inserted in both the hinges in our previous study22. This method showed great potential of detecting the food intake activity with a high accuracy without touching the skin. It was also un-obtrusive and non-intrusive, since we used a common glasses-type device.
In this study, we present a series of detailed protocols of how to implement the glasses-type device and how to use the patterns of temporalis muscle activity for monitoring the food intake and physical activity. The protocols include the process of hardware design and fabrication that consists of a 3D-printed frame of the glasses, a circuit module, and a data acquisition module, and include the software algorithms for data processing and analysis. We furthermore examined the classification among several featured activities (e.g., chewing, walking, and winking) to demonstrate the potential as a practical system that can tell a minute difference between the food intake and other physical activity patterns.

<table><tbody><tr><td>FSS1500NSB</td><td>Honeywell, USA</td><td></td><td>Load cell</td></tr><tr><td>INA125U</td><td>Texas Instruments, USA</td><td></td><td>Amplifier</td></tr><tr><td>ESP-07</td><td>Shenzhen Anxinke Technology, China</td><td></td><td>MCU with Wi-Fi module</td></tr><tr><td>74LVC1G3157</td><td>Nexperia, The Netherlands</td><td></td><td>Multiplexer</td></tr><tr><td>MP701435P</td><td>Maxpower, China</td><td></td><td>LiPo battery</td></tr><tr><td>U1V10F3</td><td>Pololu, USA</td><td></td><td>Voltage regulator</td></tr><tr><td>Ultimaker 2</td><td>Ultimaker, The Netherlands</td><td></td><td>3D printer</td></tr><tr><td>ColorFabb XT-CF20</td><td>ColorFabb, The Netherlands</td><td></td><td>Carbon fiber filament</td></tr><tr><td>iPhone 6s Plus</td><td>Apple, USA</td><td></td><td>Data acquisition device</td></tr><tr><td>Jelly Belly</td><td>Jelly Belly Candy Company, USA</td><td></td><td>Food texture for user study</td></tr></tbody></table>

design and evaluation of smart glasses for food intake and physical activity classification

This study presents a series of protocols of designing and manufacturing a glasses-type wearable device that detects the patterns of temporalis muscle activities during food intake and other physical activities. We fabricated a 3D-printed frame of the glasses and a load cell-integrated printed circuit board (PCB) module inserted in both hinges of the frame. The module was used to acquire the force signals, and transmit them wirelessly. These procedures provide the system with higher mobility, which can be evaluated in practical wearing conditions such as walking and waggling. A performance of the classification is also evaluated by distinguishing the patterns of food intake from those physical activities. A series of algorithms were used to preprocess the signals, generate feature vectors, and recognize the patterns of several featured activities (chewing and winking), and other physical activities (sedentary rest, talking, and walking). The results showed that the average F1 score of the classification among the featured activities was 91.4%. We believe this approach can be potentially useful for automatic and objective monitoring of ingestive behaviors with higher accuracy as practical means to treat ingestive problems.

Through the procedures outlined in the protocol, we prepared two versions of the 3D printed frame by differentiating the length of the head piece, LH (133 and 138 mm), and the temples, LT (110 and 125 mm), as shown in Figure 4. Therefore, we can cover several wearing conditions, which can be varied from the subjects&#39; head size, shape, etc. The subjects chose one of the frames to fit to their head for the user study. The vert...

Watch this Scientific Journal Video about Design and Evaluation of Smart Glasses for Food Intake and Physical Activity Classification at JoVE.com

Design and Evaluation of Smart Glasses for Food Intake and Physical Activity Classification

This study presents a protocol of designing and manufacturing a glasses-type wearable device that detects the patterns of food intake and other featured physical activities using load cells inserted in both hinges of the glasses.

This study presents a series of protocols of designing and manufacturing a glasses-type wearable device that detects the patterns of temporalis muscle ...

design-evaluation-smart-glasses-for-food-intake-physical-activity

Research

JoVE Journal

Engineering

11.2K Views.  Seoul National University. This study presents a protocol of designing and manufacturing a glasses-type wearable device that detects the patterns of food intake and other featured physical activities using load cells inserted in both hinges of the glasses.

Video: Design and Evaluation of Smart Glasses for Food Intake and Physical Activity Classification

Evaluation of a Smartphone-based Human Activity Recognition System in a Daily Living Environment

An evaluation method that includes continuous activities in a daily-living environment was developed for Wearable Mobility Monitoring Systems (WMMS) that attempt to recognize user activities. Participants performed a pre-determined set of daily living actions within a continuous test circuit that included mobility activities (walking, standing, sitting, lying, ascending/descending stairs), daily living tasks (combing hair, brushing teeth, preparing food, eating, washing dishes), and subtle environment changes (opening doors, using an elevator, walking on inclines, traversing staircase landings, walking outdoors). 
To evaluate WMMS performance on this circuit, fifteen able-bodied participants completed the tasks while wearing a smartphone at their right front pelvis. The WMMS application used smartphone accelerometer and gyroscope signals to classify activity states. A gold standard comparison data set was created by video-recording each trial and manually logging activity onset times. Gold standard and WMMS data were analyzed offline. Three classification sets were calculated for each circuit: (i) mobility or immobility, ii) sit, stand, lie, or walking, and (iii) sit, stand, lie, walking, climbing stairs, or small standing movement. Sensitivities, specificities, and F-Scores for activity categorization and changes-of-state were calculated.
The mobile versus immobile classification set had a sensitivity of 86.30% &#177; 7.2% and specificity of 98.96% &#177; 0.6%, while the second prediction set had a sensitivity of 88.35% &#177; 7.80% and specificity of 98.51% &#177; 0.62%. For the third classification set, sensitivity was 84.92% &#177; 6.38% and specificity was 98.17 &#177; 0.62. F1 scores for the first, second and third classification sets were 86.17 &#177; 6.3, 80.19 &#177; 6.36, and 78.42 &#177; 5.96, respectively. This demonstrates that WMMS performance depends on the evaluation protocol in addition to the algorithms. The demonstrated protocol can be used and tailored for evaluating human activity recognition systems in rehabilitation medicine where mobility monitoring may be beneficial in clinical decision-making.

A standardized evaluation method was developed for Wearable Mobility Monitoring Systems (WMMS) that includes continuous activities in a realistic daily living environment. Testing with a series of daily living activities can decrease activity recognition sensitivity; therefore, realistic testing circuits are encouraged for valid evaluation of WMMS performance.

An evaluation method that includes continuous activities in a daily-living environment was developed for Wearable Mobility Monitoring Systems (WMMS) that ...

Behavior

Control of Eating Behavior Using a Novel Feedback System

Subjects eat food from a plate that sits on a scale connected to a computer that records the weight loss of the plate during the meal and makes up a curve of food intake, meal duration and rate of eating modeled by a quadratic equation. The purpose of the method is to change eating behavior by providing visual feedback on the computer screen that the subject can adapt to because her/his own rate of eating appears on the screen during the meal. The data generated by the method is automatically analyzed and fitted to the quadratic equation using a custom made algorithm. The method has the advantage of recording eating behavior objectively and offers the possibility of changing eating behavior both in experiments and in clinical practice. A limitation may be that experimental subjects are affected by the method. The same limitation may be an advantage in clinical practice, as eating behavior is more easily stabilized by the method. A treatment that uses this method has normalized body weight and restored the health of several hundred patients with anorexia nervosa and other eating disorders and has reduced the weight and improved the health of severely overweight patients.

Subjects eat food from a plate placed on a scale connected to a computer that records the weight loss of the plate during the meal. Feedback on the computer screen allows the subject to adapt her/his eating behavior to reference curves thus normalizing body weight.

Subjects eat food from a plate that sits on a scale connected to a computer that records the weight loss of the plate during the meal and makes up a curve ...

A Novel Digital Platform for a Monitored Home-based Cardiac Rehabilitation Program

Despite the evidence that cardiac rehabilitation (CR) reduces the risk of recurrent cardiac events, only a minority of eligible patients are willing to join existing programs at cardiac rehabilitation centers. The unique remote patient monitoring system presented here enables healthcare providers to monitor CR patients at home in real-time and at low cost. The system combines mobile technology, artificial intelligence, and supportive services, expanding the delivery of medical care to the patient&#39;s home. The primary aim of the study is to increase the long-term adherence to physical activity in patients who participate in CR via the addition of a home-based digitally monitored CR component to the standard CR program in patients with ischemic heart disease (IHD), with the idea of forming new habitual health behaviors and increasing the long-term motivation for physical exercise (PE) habits at home. Secondary aims are to assess the program&#39;s impact on the physical activity level measured by average steps per day, minutes of exercise per week, blood pressure, metabolic parameters, body mass index, and waist-to-hip ratio, as well as a quality-of-life (QoL) questionnaire.The study has two arms: (1) home-based monitored exercise using a smart digital garment and wristband, in addition to motivation and reinforcement via text messages; (2) standard CR facility-based exercise. The study design is a randomized, controlled trial comparing standard CR to a home-based monitoring and reinforcement program. The study program is designed for 12 weeks.Clinical tests and anthropometric measurements are performed before and after the study, measuring height, weight, waist circumference, visceral fat and body mass index (BMI), blood pressure, and HbA1c and lipid profile. Patients have to complete a baseline survey including socio-demographic characteristics and QoL questionnaire SF-36. At the end of the study, patients complete a survey regarding the use of the smart digital garment&#39;s benefits and usability. The study is currently underway.

The aim is to promote a new approach to cardiac rehabilitation (CR), using a unique remote patient monitoring system that will enable healthcare providers to monitor CR patients at home at low cost, with the intention of making CR services more accessible and improving compliance. The study is currently underway.

Despite the evidence that cardiac rehabilitation (CR) reduces the risk of recurrent cardiac events, only a minority of eligible patients are willing to ...

A Methodology for Capturing Joint Visual Attention Using Mobile Eye-Trackers

With the advent of new technological advances, it is possible to study social interactions at a microlevel with unprecedented accuracy. High frequency sensors, such as eye-trackers, electrodermal activity wristbands, EEG bands, and motion sensors provide observations at the millisecond level. This level of precision allows researchers to collect large datasets on social interactions. In this paper, I discuss how multiple eye-trackers can capture a fundamental construct in social interactions, joint visual attention (JVA). JVA has been studied by developmental psychologists to understand how children acquire language, learning scientists to understand how small groups of learners work together, and social scientists to understand interactions in small teams. This paper describes a methodology for capturing JVA in colocated settings using mobile eye-trackers. It presents some empirical results and discusses implications of capturing microobservations to understand social interactions.

Using multimodal sensors is a promising way to understand the role of social interactions in educational settings. This paper describes a methodology for capturing joint visual attention from colocated dyads using mobile eye-trackers.

With the advent of new technological advances, it is possible to study social interactions at a microlevel with unprecedented accuracy. High frequency ...

Iterative Development of an Innovative Smartphone-Based Dietary Assessment Tool: Traqq

To collect dietary intake data in a fast and reliable manner, a flexible and innovative smartphone application (app) called Traqq was developed (iOS/Android). This app can be used as a food record and&#160;24-h recall (or shorter recall periods). Different sampling schemes can be created on either prespecified or random days/times within a predetermined period for both methods, with&#160;push notifications to urge the participants to register their food intake. In case of non-response, notifications are automatically rescheduled to ensure complete data collection. For use as a food record, respondents can access the app and log their food intake throughout the day. Food records close automatically at the end of the day; recalls close after submission of the consumed items. The recall as well as the food record module provide access to an extensive food list based on the Dutch food composition database (FCDB), which can be accustomed to fit different research purposes. When selecting a food item, respondents are simultaneously prompted to insert portion size, i.e., in household measures (e.g., cups, spoons, glasses), standard portion sizes (e.g., small, medium, large), or weight in grams, and eating occasion/time of consumption. Portion size options can be adjusted, e.g., only entry in grams in case of a weighed food record or time of consumption instead of eating occasion). The app also includes a My Dishes function, which allows the respondent to create their own recipes or product combinations (e.g., a daily breakfast) and only report the total quantity consumed. Subsequently, the app accounts for yield and retention factors. The data are stored on a secure server. If desired, additional questions, i.e., in general or those related to specific food items or eating occasions can be incorporated. This paper describes the development of the system (app and backend), including expert evaluations and usability testing.

This article describes the protocol for the development of an innovative smartphone-based dietary assessment application Traqq, including expert evaluations and usability testing.

To collect dietary intake data in a fast and reliable manner, a flexible and innovative smartphone application (app) called Traqq was developed ...

Concept Development and Use of an Automated Food Intake and Eating Behavior Assessment Method

The vast majority of dietary and eating behavior assessment methods are based on self-reports. They are burdensome and also prone to measurement errors. Recent technological innovations allow for the development of more accurate and precise dietary and eating behavior assessment tools that require less effort for both the user and the researcher. Therefore, a new sensor-based device to assess food intake and eating behavior was developed. The device is a regular dining tray equipped with a video camera and three separate built-in weighing stations. The weighing stations measure the weight of the bowl, plate, and drinking cup continuously over the course of a meal. The video camera positioned to the face records eating behavior characteristics (chews, bites), which are analyzed using artificial intelligence (AI)-based automatic facial expression software. The tray weight and the video data are transported at real-time to a personal computer (PC) using a wireless receiver. The outcomes of interest, such as the amount eaten, eating rate and bite size, can be calculated by subtracting the data of these measures at the timepoints of interest. The information obtained by the current version of the tray can be used for research purposes, an upgraded version of the device would also facilitate the provision of more personalized advice on dietary intake and eating behavior. Contrary to the conventional dietary assessment methods, this dietary assessment device measures food intake directly within a meal and is not dependent on memory or the portion size estimation. Ultimately, this device is therefore suited for daily main meal food intake and eating behavior measures. In the future, this technology based dietary assessment method can be linked to health applications or smart watches to obtain a complete overview of exercise, energy intake, and eating behavior.

This protocol shows and explains a new technology-based dietary assessment method. The method consists of a dining tray with multiple built-in weighing scales and a video camera. The device is unique in the sense that it incorporates automated measures of food and drink intake and eating behavior over the course of a meal.

The vast majority of dietary and eating behavior assessment methods are based on self-reports. They are burdensome and also prone to measurement errors. ...

Evaluating Usability Aspects of a Mixed Reality Solution for Immersive Analytics in Industry 4.0 Scenarios

In medicine or industry, the analysis of high-dimensional data sets is increasingly required. However, available technical solutions are often complex to use. Therefore, new approaches like immersive analytics are welcome. Immersive analytics promise to experience high-dimensional data sets in a convenient manner for various user groups and data sets. Technically, virtual-reality devices are used to enable immersive analytics. In Industry 4.0, for example, scenarios like the identification of outliers or anomalies in high-dimensional data sets are pursued goals of immersive analytics. In this context, two important questions should be addressed for any developed technical solution on immersive analytics: First, is the technical solutions being helpful or not? Second, is the bodily experience of the technical solution positive or negative? The first question aims at the general feasibility of a technical solution, while the second one aims at the wearing comfort. Extant studies and protocols, which systematically address these questions are still rare. In this work, a study protocol is presented, which mainly investigates the usability for immersive analytics in Industry 4.0 scenarios. Specifically, the protocol is based on four pillars. First, it categorizes users based on previous experiences. Second, tasks are presented, which can be used to evaluate the feasibility of the technical solution. Third, measures are presented, which quantify the learning effect of a user. Fourth, a questionnaire&#160;evaluates the stress level when performing tasks. Based on these pillars, a technical setting was implemented that uses mixed reality smartglasses to apply the study protocol. The results of the conducted study show the applicability of the protocol on the one hand and the feasibility of immersive analytics in Industry 4.0 scenarios on the other. The presented protocol includes a discussion of discovered limitations.

This protocol delineates the technical setting of a developed mixed reality application that is used for immersive analytics. Based on this, measures are presented, which were used in a study to gain insights into usability aspects of the developed technical solution.

In medicine or industry, the analysis of high-dimensional data sets is increasingly required. However, available technical solutions are often complex to ...

Deep Neural Networks for Image-Based Dietary Assessment

Due to the issues and costs associated with manual dietary assessment approaches, automated solutions are required to ease and speed up the work and increase its quality. Today, automated solutions are able to record a person's dietary intake in a much simpler way, such as by taking an image with a smartphone camera. In this article, we will focus on such image-based approaches to dietary assessment. For the food image recognition problem, deep neural networks have achieved the state of the art in recent years, and we present our work in this field. In particular, we first describe the method for food and beverage image recognition using a deep neural network architecture, called NutriNet. This method, like most research done in the early days of deep learning-based food image recognition, is limited to one output per image, and therefore unsuitable for images with multiple food or beverage items. That is why approaches that perform food image segmentation are considerably more robust, as they are able to identify any number of food or beverage items in the image. We therefore also present two methods for food image segmentation - one is based on fully convolutional networks (FCNs), and the other on deep residual networks (ResNet).

The goal of the work presented in this article is to develop technology for automated recognition of food and beverage items from images taken by mobile devices. The technology comprises of two different approaches - the first one performs food image recognition while the second one performs food image segmentation.

Due to the issues and costs associated with manual dietary assessment approaches, automated solutions are required to ease and speed up the work and ...

A Single-Channel and Non-Invasive Wearable Brain-Computer Interface for Industry and Healthcare

The present work focuses on how to build a wearable brain-computer interface (BCI). BCIs are a novel means of human-computer interaction that relies on direct measurements of brain signals to assist both people with disabilities and those who are able-bodied. Application examples include robotic control, industrial inspection, and neurorehabilitation. Notably, recent studies have shown that steady-state visually evoked potentials (SSVEPs) are particularly suited for communication and control applications, and efforts are currently being made to bring BCI technology into daily life. To achieve this aim, the final system must rely on wearable, portable, and low-cost instrumentation. In exploiting SSVEPs, a flickering visual stimulus with fixed frequencies is required. Thus, in considering daily-life constraints, the possibility to provide visual stimuli by means of smart glasses was explored in this study. Moreover, to detect the elicited potentials, a commercial device for electroencephalography (EEG) was considered. This consists of a single differential channel with dry electrodes (no conductive gel), thus achieving the utmost wearability and portability. In such a BCI, the user can interact with the smart glasses by merely staring at icons appearing on the display. Upon this simple principle, a user-friendly, low-cost BCI was built by integrating extended reality (XR) glasses with a commercially available EEG device. The functionality of this wearable XR-BCI was examined with an experimental campaign involving 20 subjects. The classification accuracy was between 80%-95% on average depending on the stimulation time. Given these results, the system can be used as a human-machine interface for industrial inspection but also for rehabilitation in ADHD and autism.

This paper discusses how to build a brain-computer interface by relying on consumer-grade equipment and steady-state visually evoked potentials. For this, a single-channel electroencephalograph exploiting dry electrodes was integrated with augmented reality glasses for stimuli presentation and output data visualization. The final system was non-invasive, wearable, and portable.

The present work focuses on how to build a wearable brain-computer interface (BCI). BCIs are a novel means of human-computer interaction that relies on ...

Enhancing Classroom Engagement: AI-Based Analysis of Student Attention Levels

Artificial Intelligence-Based System for Detecting Attention Levels in Students

The attention level of students in a classroom can be improved through the use of Artificial Intelligence (AI) techniques. By automatically identifying the attention level, teachers can employ strategies to regain students' focus. This can be achieved through various sources of information.
One source is to analyze the emotions reflected on students' faces. AI can detect emotions, such as neutral, disgust, surprise, sadness, fear, happiness, and anger. Additionally, the direction of the students' gaze can also potentially indicate their level of attention. Another source is to observe the students' body posture. By using cameras and deep learning techniques, posture can be analyzed to determine the level of attention. For example, students who are slouching or resting their heads on their desks may have a lower level of attention. Smartwatches distributed to the students can provide biometric and other data, including heart rate and inertial measurements, which can also be used as indicators of attention. By combining these sources of information, an AI system can be trained to identify the level of attention in the classroom. However, integrating the different types of data poses a challenge that requires creating a labeled dataset. Expert input and existing studies are consulted for accurate labeling. In this paper, we propose the integration of such measurements and the creation of a dataset and a potential attention classifier. To provide feedback to the teacher, we explore various methods, such as smartwatches or direct computers. Once the teacher becomes aware of attention issues, they can adjust their teaching approach to re-engage and motivate the students. In summary, AI techniques can automatically identify the students' attention level by analyzing their emotions, gaze direction, body posture, and biometric data. This information can assist teachers in optimizing the teaching-learning process.

Author Spotlight: Addressing Technical and Subjective Challenges in Measuring Classroom Attention

This paper proposes an artificial intelligence-based system to automatically detect whether students are paying attention to the class or are distracted. This system is designed to help teachers maintain students' attention, optimize their lessons, and dynamically introduce modifications in order for them to be more engaging.

The attention level of students in a classroom can be improved through the use of Artificial Intelligence (AI) techniques. By automatically identifying ...