This work was developed with funding from Programa Prometeo, project ID CIPROM/2021/017. Prof. Rosabel Roig is the chair of the UNESCO &#34;Education, Research and Digital Inclusion&#34;.

The following protocol follows the guidelines of the University of Alicante&#39;s human research ethics committee with the approved protocol number UA-2022-11-12. Informed consent has been obtained from all participants for this experiment and for using the data here.
1. Hardware, software, and class setup
<ol>
	<li>Set a router with WiFi capabilities (the experiments were carried out using a DLink DSR 1000AC) in the desired location so that its range covers the entire room. Here, 25 m2 classrooms with 30 students were covered.</li>
	<li>Set one smartwatch (here Samsung Galaxy Smartwatch 5) and one camera (here Logitech C920 cameras) for each student location. Set one embedded device for every two students. Fix two cameras on two tripods and connect them to another embedded device (hereafter referred to as zenithal cameras).</li>
	<li>Connect the cameras to the corresponding embedded devices with a USB link and turn them ON. Turn ON the smartwatches as well. Connect every embedded device and smartwatch to the WiFi network of the router set up in step 1.1.</li>
	<li>Place the cameras in the corner of each student&#39;s desk and point them forward and slightly tilted upwards so that the face of the student is clearly visible in the images.</li>
	<li>Place the two tripods with the cameras in front of the desks that are closest to the aisles in the first row of seats of the classroom. Move the tripod riser to the highest position so the cameras can clearly view most of the students. Each embedded device will be able to manage one or two students, along with their respective cameras and watches. The setup is depicted in Figure 1.</li>
	<li>Run the capturing software in the smartwatches and in the embedded devices so that they are ready to send the images and the accelerometer, gyroscope, heart rate, and lighting data. Run the server software that collects the data and stores it. A diagram of all these elements can be seen in Figure 2.</li>
	<li>Ensure the zenithal cameras dominate the scene so that they have a clear view of the students&#39; bodies. Place an additional individual camera in front of each student.</li>
	<li>Let the students sit in their places, inform them of the goals of the experiment, and instruct them to wear the smartwatches on their dominant hand and not to interact with any of the elements of the setup.</li>
	<li>Start the capture and collection of the data in the server and resume the class lessons as usual.</li>
</ol>
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/65931/65931fig01.jpg" /> 
Figure 1: Hardware and data pipeline. The cameras and smartwatches data are gathered and fed to the machine learning algorithms to be processed. <a href="https://www.jove.com/files/ftp_upload/65931/65931fig01large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/65931/65931fig02.jpg" /> 
Figure 2: Position of the sensors, teacher, and students. Diagram showing the positions of the cameras, smartwatches, and GUI in the classroom with the teacher and students. <a href="https://www.jove.com/files/ftp_upload/65931/65931fig02large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
2. Capture and data processing pipeline
NOTE: All these steps are performed automatically by processing software deployed in a server. The implementation used for the experiments in this work was written in Python 3.8.
<ol>
	<li>Collect the required data by gathering all the images and biometric data from the smartwatch for each student and build a data frame that includes data from 1 s. This data frame is composed of one image from the individual camera, one image from each zenithal camera (two in this case), fifty registers of the three gyroscope values, fifty registers of the three accelerometer values, one heart-rate value, and one light-condition value.</li>
	<li>To compute the head direction, the first step is to use the webcam that is pointed to the student&#39;s face and retrieve an image. Then, that image is processed by the BlazeFace17 landmark estimation algorithm. The output of such an algorithm is a list of 2D key points corresponding to specific areas of the face.</li>
	<li>By using the estimated position of the eyes, nose, mouth, and chin provided by the algorithm, solve the perspective n-point problem using the method cv::SolvePnPMethod of the OpenCV library, a canonical face setting. The rotation matrix that defines the head direction is obtained as a result of this procedure.</li>
	<li>Compute the body pose by forwarding the zenithal image from the data frame in which the student is depicted to the BlazePose18 landmark estimation algorithm and retrieve a list of 2D coordinates of the joints of the body depicted in the image. This list of landmarks describes the student&#39;s pose. 
	NOTE: The body pose is important as it can accurately represent a student&#39;s engagement with different activities during the class. For instance, if the student is sitting naturally with their hands on the desk, it could mean that they are taking notes. In contrast, if they are continually moving their arms, it could mean they are speaking to someone.</li>
	<li>Obtain the image of the student&#39;s face and perform alignment preprocessing. Retrieve the key points of the eyes from the list computed in step 2.3 and project them to the image plane so that the position of the eyes is obtained.</li>
	<li>Trace a virtual vector that joins the position of both eyes. Compute the angle between that vector and the horizontal line and build a homography matrix to apply the reverse angle to the image and the key points so that the eyes are aligned horizontally.</li>
	<li>Crop the face using the key points to detect its limits and feed the patch to the emotion detection convolutional neural network19. Retrieve the output from the network, which is a vector with 7 positions, each position giving the probability of the face showing one of the seven basic emotions: neutral, happy, disgusted, angry, scared, surprised, and sad.</li>
	<li>Measure the level of attention by obtaining the accelerometer, gyroscope, and heart rate data from the data frame (Figure 3).</li>
	<li>Feed the deep learning-based model that was custom-built and trained from scratch described in the representative results with the stream of accelerometer and gyroscope data. Retrieve the output from the model, which is a vector with 4 positions, each position giving the probability of the data representing one of the following possible actions: handwriting, typing on a keyboard, using a mobile phone, or resting, as shown in Figure 4.</li>
	<li>Feed the final attention classifier with a linearization of all the results from the previous systems by merging the head direction, emotion recognition output, body pose, and the gyroscope, accelerometer, and heart rate data. Retrieve the results from this final classifier, which is a score from 0 to 100. Classify this continuous value into one of the three possible discrete categories of attention: low level of attention (0-40%), medium level of attention (40%-75%), and high level of attention (75% - 100%). The structure of the dataset generated is shown in Table 1.</li>
	<li>Show the results of the attention levels to the teacher using the graphical user interface (GUI) from the teacher&#39;s computer, accessible from a regular web browser.</li>
</ol>
<img alt="Figure 3" class="xfigimg" src="/files/ftp_upload/65931/65931fig03.jpg" /> 
Figure 3: Data captured by the smartwatch. The smartwatch provides a gyroscope, accelerometer, heart rate, and light condition as streams of data. <a href="https://www.jove.com/files/ftp_upload/65931/65931fig03large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 4" class="xfigimg" src="/files/ftp_upload/65931/65931fig04.jpg" /> 
Figure 4: Examples of the categories considered by the activity recognition model. Four different actions are recognized by the activity recognition model: handwriting, typing on a keyboard, using a smartphone, and resting position. <a href="https://www.jove.com/files/ftp_upload/65931/65931fig04large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>

Enhancing Classroom Engagement: AI-Based Analysis of Student Attention Levels

<ol>
	<li>Hasnine, M. N., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Students'+emotion+extraction+and+visualization+for+engagement+detection+in+online+learning.">Students' emotion extraction and visualization for engagement detection in online learning.</a> Procedia Comp Sci. 192, 3423-3431 (2021).</li><li>Khare, S. K., Blanes-Vidal, V., Nadimi, E. S., Acharya, U. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Emotion+recognition+and+artificial+intelligence:+A+systematic+review+(2014-2023)+and+research+recommendations.">Emotion recognition and artificial intelligence: A systematic review (2014-2023) and research recommendations.</a> Info Fusion. 102, 102019 (2024).</li><li>Bosch, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Detecting+student+engagement:+Human+versus+machine.">Detecting student engagement: Human versus machine.</a> UMAP '16: Proc the 2016 Conf User Model Adapt Personal. , 317-320 (2016).</li><li>Araya, R., Sossa-Rivera, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automatic+detection+of+gaze+and+body+orientation+in+elementary+school+classrooms.">Automatic detection of gaze and body orientation in elementary school classrooms.</a> Front Robot AI. 8, 729832 (2021).</li><li>Lu, Y., Zhang, J., Li, B., Chen, P., Zhuang, Z. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Harnessing+commodity+wearable+devices+for+capturing+learner+engagement.">Harnessing commodity wearable devices for capturing learner engagement.</a> IEEE Access. 7, 15749-15757 (2019).</li><li>Vanneste, P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Computer+vision+and+human+behaviour,+emotion+and+cognition+detection:+A+use+case+on+student+engagement.">Computer vision and human behaviour, emotion and cognition detection: A use case on student engagement.</a> Mathematics. 9 (3), 287 (2021).</li><li>Ma, X., Xu, M., Dong, Y., Sun, Z. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automatic+student+engagement+in+online+learning+environment+based+on+neural+Turing+machine.">Automatic student engagement in online learning environment based on neural Turing machine.</a> Int J Info Edu Tech. 11 (3), 107-111 (2021).</li><li>Wood, E., Bulling, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=EyeTab:+model-based+gaze+estimation+on+unmodified+tablet+computers.">EyeTab: model-based gaze estimation on unmodified tablet computers.</a> ETRA '14: Proc Symp Eye Tracking Res Appl. , 207-210 (2014).</li><li>Sanghvi, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automatic+analysis+of+affective+postures+and+body+motion+to+detect+engagement+with+a+game+companion.">Automatic analysis of affective postures and body motion to detect engagement with a game companion.</a> HRI '11: Proc 6th Int Conf Human-robot Interact. , 205-211 (2011).</li><li>Gupta, S., Kumar, P., Tekchandani, R. K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Facial+emotion+recognition+based+real-time+learner+engagement+detection+system+in+online+learning+context+using+deep+learning+models.">Facial emotion recognition based real-time learner engagement detection system in online learning context using deep learning models.</a> Multimed Tools Appl. 82 (8), 11365-11394 (2023).</li><li>Altuwairqi, K., Jarraya, S. K., Allinjawi, A., Hammami, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Student+behavior+analysis+to+measure+engagement+levels+in+online+learning+environments.">Student behavior analysis to measure engagement levels in online learning environments.</a> Signal Image Video Process. 15 (7), 1387-1395 (2021).</li><li>Belle, A., Hargraves, R. H., Najarian, K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=An+automated+optimal+engagement+and+attention+detection+system+using+electrocardiogram.">An automated optimal engagement and attention detection system using electrocardiogram.</a> Comput Math Methods Med. 2012, 528781 (2012).</li><li>Alban, A. Q., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Heart+rate+as+a+predictor+of+challenging+behaviours+among+children+with+autism+from+wearable+sensors+in+social+robot+interactions.">Heart rate as a predictor of challenging behaviours among children with autism from wearable sensors in social robot interactions.</a> Robotics. 12 (2), 55 (2023).</li><li>Kajiwara, Y., Shimauchi, T., Kimura, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Predicting+emotion+and+engagement+of+workers+in+order+picking+based+on+behavior+and+pulse+waves+acquired+by+wearable+devices.">Predicting emotion and engagement of workers in order picking based on behavior and pulse waves acquired by wearable devices.</a> Sensors. 19 (1), 165 (2019).</li><li>Costante, G., Porzi, L., Lanz, O., Valigi, P., Ricci, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Personalizing+a+smartwatch-based+gesture+interface+with+transfer+learning.">Personalizing a smartwatch-based gesture interface with transfer learning.</a> , 2530-2534 (2014).</li><li>Mekruksavanich, S., Jitpattanakul, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Deep+convolutional+neural+network+with+RNNs+for+complex+activity+recognition+using+wrist-worn+wearable+sensor+data.">Deep convolutional neural network with RNNs for complex activity recognition using wrist-worn wearable sensor data.</a> Electronics. 10 (14), 1685 (2021).</li><li>Bazarevsky, V., Kartynnik, Y., Vakunov, A., Raveendran, K., Grundmann, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=BlazeFace:+Sub-millisecond+Neural+Face+Detection+on+Mobile+GPUs.">BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs.</a> arXiv. , (2019).</li><li>Bazarevsky, V., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=BlazePose:+On-device+Real-time+Body+Pose+tracking.">BlazePose: On-device Real-time Body Pose tracking.</a> arXiv. , (2020).</li><li>Mejia-Escobar, C., Cazorla, M., Martinez-Martin, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Towards+a+better+performance+in+facial+expression+recognition:+a+data-centric+approach.">Towards a better performance in facial expression recognition: a data-centric approach.</a> Comput Intelligence Neurosci. , (2023).</li><li>El-Garem, A., Adel, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Applying+systematic+literature+review+and+Delphi+methods+to+explore+digital+transformation+key+success+factors.">Applying systematic literature review and Delphi methods to explore digital transformation key success factors.</a> Int J Eco Mgmt Engi. 16 (7), 383-389 (2022).</li><li>Indumathi, V., Kist, A. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Using+electroencephalography+to+determine+student+attention+in+the+classroom.">Using electroencephalography to determine student attention in the classroom.</a> , 1-3 (2023).</li><li>Ma, X., Xie, Y., Wang, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Research+on+the+construction+and+application+of+teacher-student+interaction+evaluation+system+for+smart+classroom+in+the+post+COVID-19.">Research on the construction and application of teacher-student interaction evaluation system for smart classroom in the post COVID-19.</a> Studies Edu Eval. 78, 101286 (2023).</li><li>Andersen, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Constructing+Delphi+statements+for+technology+foresight.">Constructing Delphi statements for technology foresight.</a> Futures Foresight Sci. 5 (2), e144 (2022).</li><li>Khodyakov, D., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Disciplinary+trends+in+the+use+of+the+Delphi+method:+A+bibliometric+analysis.">Disciplinary trends in the use of the Delphi method: A bibliometric analysis.</a> PLoS One. 18 (8), e0289009 (2023).</li><li>Martins, A. I., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Consensus+on+the+Terms+and+Procedures+for+Planning+and+Reporting+a+Usability+Evaluation+of+Health-Related+Digital+Solutions:+Delphi+Study+and+a+Resulting+Checklist.">Consensus on the Terms and Procedures for Planning and Reporting a Usability Evaluation of Health-Related Digital Solutions: Delphi Study and a Resulting Checklist.</a> J Medical Internet Res. 25, e44326 (2023).</li><li>Dalmaso, M., Castelli, L., Galfano, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Social+modulators+of+gaze-mediated+orienting+of+attention:+A+review.">Social modulators of gaze-mediated orienting of attention: A review.</a> Psychon Bull Rev. 27 (5), 833-855 (2020).</li><li>Klein, R. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Thinking+about+attention:+Successive+approximations+to+a+productive+taxonomy.">Thinking about attention: Successive approximations to a productive taxonomy.</a> Cognition. 225, 105137 (2022).</li><li>Schindler, S., Bublatzky, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Attention+and+emotion:+An+integrative+review+of+emotional+face+processing+as+a+function+of+attention.">Attention and emotion: An integrative review of emotional face processing as a function of attention.</a> Cortex. 130, 362-386 (2020).</li><li>Zaletelj, J., Ko&#353;ir, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Predicting+students'+attention+in+the+classroom+from+Kinect+facial+and+body+features.">Predicting students' attention in the classroom from Kinect facial and body features.</a> J Image Video Proc. 80, (2017).</li><li>Strauch, C., Wang, C. A., Einh&#228;user, W., Van der Stigchel, S., Naber, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Pupillometry+as+an+integrated+readout+of+distinct+attentional+networks.">Pupillometry as an integrated readout of distinct attentional networks.</a> Trends Neurosci. 45 (8), 635-647 (2022).</li></ol>

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

This work presents a system that measures the attention level of a student in a classroom using cameras, smartwatches, and artificial intelligence algorithms. This information is subsequently presented to the teacher for them to have an idea of the general state of the class.
One of the main critical steps of the protocol is the synchronization of the smartwatch information with the color camera image, as these have different frequencies. This was solved by deploying raspberries as servers tha...

In modern educational settings, accurately assessing and maintaining students&#39; attention is crucial for effective teaching and learning. However, traditional methods of gauging engagement, such as self-reporting or subjective teacher observations, are time-consuming and prone to biases. To address this challenge, Artificial Intelligence (AI) techniques have emerged as promising solutions for automated attention detection. One significant aspect of understanding students&#39; engagement levels is emotion recognition1. AI systems can analyze facial expressions to identify emotions, such as neutral, disgust, surprise, sadness, fear, happiness, and anger2.
Gaze direction and body posture are also crucial indicators of students&#39; attention3. By utilizing cameras and advanced machine learning algorithms, AI systems can accurately track where students are looking and analyze their body posture to detect signs of disinterest or fatigue4. Furthermore, incorporating biometric data enhances the accuracy and reliability of attention detection5. By collecting measurements, such as heart rate and blood oxygen saturation levels, through smartwatches worn by students, objective indicators of attention can be obtained, complementing other sources of information.
This paper proposes a system that evaluates an individual&#39;s level of attention using color cameras and other different sensors. It combines emotion recognition, gaze direction analysis, body posture assessment, and biometric data to provide educators with a comprehensive set of tools for optimizing the teaching-learning process and improving student engagement. By employing these tools, educators can gain a comprehensive understanding of the teaching-learning process and enhance student engagement, thereby optimizing the overall educational experience. By applying AI techniques, it is even possible to automatically evaluate this data.
The main goal of this work is to describe the system that allows us to capture all the information and, once captured, to train an AI model that allows us to obtain the attention of the whole class in real-time. Although other works have already proposed capturing attention using visual or emotional information6, this work proposes the combined use of these techniques, which provides a holistic approach to allow the use of more complex and effective AI techniques. Moreover, the datasets hitherto available are limited to either a set of videos or one of biometric data. The literature includes no datasets that provide complete data with images of the student&#39;s face or their body, biometric data, data on the teacher&#39;s position, etc. With the system presented here, it is possible to capture this type of dataset.
The system associates a level of attention with each student at each point of time. This value is a probability value of attention between 0% and 100%, which can be interpreted as low level of attention (0%-40%), medium level of attention (40%-75%), and high level of attention (75%-100%). Throughout the text, this probability of attention is referred to as the level of attention, student attention, or whether students are distracted or not, but these are all related to the same output value of our system.
Over the years, the field of automatic engagement detection has grown significantly due to its potential to revolutionize education. Researchers have proposed various approaches for this area of study.
Ma et al.7 introduced a novel method based on a Neural Turing Machine for automatic engagement recognition. They extracted certain features, such as eye gaze, facial action units, head pose, and body pose, to create a comprehensive representation of engagement recognition.
EyeTab8, another innovative system, used models to estimate where someone is looking with both their eyes. It was specially made to work smoothly on a standard tablet with no modifications. This system harnesses well-known algorithms for processing images and analyzing computer vision. Their gaze estimation pipeline includes a Haar-like feature-based eye detector, as well as a RANSAC-based limbus ellipse fitting approach.
Sanghvi et al.9 propose an approach that relies on vision-based techniques to automatically extract expressive postural features from videos recorded from a lateral view, capturing the behavior of the children. An initial evaluation is conducted, involving the training of multiple recognition models using contextualized affective postural expressions. The results obtained demonstrate that patterns of postural behavior can effectively predict the engagement of the children with the robot.
In other works, such as&#160;Gupta et al.10, a deep learning-based method is employed to detect the real-time engagement of online learners by analyzing their facial expressions and classifying their emotions. The approach utilizes facial emotion recognition to calculate an engagement index (EI) that predicts two engagement states: engaged and disengaged. Various deep learning models, including Inception-V3, VGG19, and ResNet-50, are evaluated and compared to identify the most effective predictive classification model for real-time engagement detection.
In&#160;Altuwairqi et al.11, the researchers present a novel automatic multimodal approach for assessing student engagement levels in real-time. To ensure accurate and dependable measurements, the team integrated and analyzed three distinct modalities that capture students&#39; behaviors: facial expressions for emotions, keyboard keystrokes, and mouse movements.
Guill&#233;n et al.12 propose the development of a monitoring system that uses electrocardiography (ECG) as a primary physiological signal to analyze and predict the presence or absence of cognitive attention in individuals while performing a task.
Alban et al.13 utilize a neural network (NN) to detect emotions by analyzing the heart rate (HR) and electrodermal activity (EDA) values of various participants in both time and frequency domains. They find that an increase in the root-mean-square of successive differences (RMSDD) and the standard deviation normal-to-normal (SDNN) intervals, coupled with a decrease in the average HR, indicate heightened activity in the sympathetic nervous system, which is associated with fear.
Kajiwara et al.14 propose an innovative system that employs wearable sensors and deep neural networks to forecast the level of emotion and engagement in workers. The system follows a three-step process. Initially, wearable sensors capture and collect data on behaviors and pulse waves. Subsequently, time series features are computed based on the behavioral and physiological data acquired. Finally, deep neural networks are used to input the time series features and make predictions on the individual&#39;s emotions and engagement levels.
In other research, such as&#160;Costante et al.15, an approach based on a novel transfer metric learning algorithm is proposed, which utilizes prior knowledge of a predefined set of gestures to enhance the recognition of user-defined gestures. This improvement is achieved with minimal reliance on additional training samples. Similarly, a sensor-based human activity recognition framework16 is presented to address the goal of the impersonal recognition of complex human activities. Signal data collected from wrist-worn sensors is utilized in the human activity recognition framework developed, employing four RNN-based DL models (Long-Short Term Memories, Bidirectional Long-Short Term Memories, Gated Recurrent Units, and Bidirectional Gated Recurrent Units) to investigate the activities performed by the user of the wearable device.

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>4 GPUs&#160; Nvidia A40 Ampere</td>
			<td>NVIDIA</td>
			<td>TCSA40M-PB</td>
			<td>GPU for centralized model processing server</td>
		</tr>
		<tr>
			<td>FusionServer 2288H V5</td>
			<td>X-Fusion</td>
			<td>02311XBK</td>
			<td>Platform that includes power supply and motherboard for centralized model processing server</td>
		</tr>
		<tr>
			<td>Memory Card Evo Plus 128 GB</td>
			<td>Samsung</td>
			<td>MB-MC128KA/EU</td>
			<td>Memory card for the operation of the raspberry pi 4b 2gb.&#160; One for each raspberry.&#160;</td>
		</tr>
		<tr>
			<td>NEMIX RAM - 512 GB Kit DDR4-3200 PC4-25600 8Rx4 EC</td>
			<td>NEMIX</td>
			<td>M393AAG40M32-CAE</td>
			<td>RAM for centralized model processing server</td>
		</tr>
		<tr>
			<td>Processor Intel Xeon Gold 6330</td>
			<td>Intel</td>
			<td>CD8068904572101</td>
			<td>Processor for centralized model processing server</td>
		</tr>
		<tr>
			<td>Raspberry PI 4B 2GB</td>
			<td>Raspberry</td>
			<td>1822095</td>
			<td>Local server that receives requests from the clocks and sends them to the general server. One every two students.</td>
		</tr>
		<tr>
			<td>Samsung Galaxy Watch 5 (40mm)</td>
			<td>Samsung</td>
			<td>SM-R900NZAAPHE</td>
			<td>Clock that monitors each student&#39;s activity. For each student.&#160;</td>
		</tr>
		<tr>
			<td>Samsung MZQL23T8HCLS-00B7C PM9A3 3.84Tb Nvme U.2 PCI-Express-4 x4 2.5inch Ssd</td>
			<td>Samsung</td>
			<td>MZQL23T8HCLS-00B7C</td>
			<td>Internal storage for centralized model processing server</td>
		</tr>
		<tr>
			<td>WebCam HD Pro C920 Webcam FullHD</td>
			<td>Logitech</td>
			<td>960-001055</td>
			<td>Webcam HD. One for each student plus two for student poses.</td>
		</tr>
	</tbody>
</table>

artificial intelligence-based system for detecting attention levels in students

The attention level of students in a classroom can be improved through the use of Artificial Intelligence (AI) techniques. By automatically identifying the attention level, teachers can employ strategies to regain students' focus. This can be achieved through various sources of information.
One source is to analyze the emotions reflected on students' faces. AI can detect emotions, such as neutral, disgust, surprise, sadness, fear, happiness, and anger. Additionally, the direction of the students' gaze can also potentially indicate their level of attention. Another source is to observe the students' body posture. By using cameras and deep learning techniques, posture can be analyzed to determine the level of attention. For example, students who are slouching or resting their heads on their desks may have a lower level of attention. Smartwatches distributed to the students can provide biometric and other data, including heart rate and inertial measurements, which can also be used as indicators of attention. By combining these sources of information, an AI system can be trained to identify the level of attention in the classroom. However, integrating the different types of data poses a challenge that requires creating a labeled dataset. Expert input and existing studies are consulted for accurate labeling. In this paper, we propose the integration of such measurements and the creation of a dataset and a potential attention classifier. To provide feedback to the teacher, we explore various methods, such as smartwatches or direct computers. Once the teacher becomes aware of attention issues, they can adjust their teaching approach to re-engage and motivate the students. In summary, AI techniques can automatically identify the students' attention level by analyzing their emotions, gaze direction, body posture, and biometric data. This information can assist teachers in optimizing the teaching-learning process.

The target group of this study is undergraduate and master&#39;s students, and so the main age group is between 18 and 25 years old. This population was selected because they can handle electronic devices with fewer distractions than younger students. In total, the group included 25 people. This age group can provide the most reliable results to test the proposal.
The results of the attention level shown to the teacher have 2 parts. Part A of the result shows individual information about the c...

Read this Scientific Article Protocol about Author Spotlight: Addressing Technical and Subjective Challenges in Measuring Classroom Attention at JoVE.com

Author Spotlight: Addressing Technical and Subjective Challenges in Measuring Classroom Attention

This paper proposes an artificial intelligence-based system to automatically detect whether students are paying attention to the class or are distracted. This system is designed to help teachers maintain students' attention, optimize their lessons, and dynamically introduce modifications in order for them to be more engaging.

Artificial Intelligence-Based System for Detecting Attention Levels in Students

The attention level of students in a classroom can be improved through the use of Artificial Intelligence (AI) techniques. By automatically identifying ...

artificial-intelligence-based-system-for-detecting-attention-levels

Research

JoVE Journal

Engineering

2.1K Views.  University of Alicante. This paper proposes an artificial intelligence-based system to automatically detect whether students are paying attention to the class or are distracted. This system is designed to help teachers maintain students' attention, optimize their lessons, and dynamically introduce modifications in order for them to be more engaging.

Video: Author Spotlight: Addressing Technical and Subjective Challenges in Measuring Classroom Attention

Light Enhanced Hydrofluoric Acid Passivation: A Sensitive Technique for Detecting Bulk Silicon Defects

A procedure to measure the bulk lifetime (&#62;100 &#181;sec) of silicon wafers by temporarily attaining a very high level of surface passivation when immersing the wafers in hydrofluoric acid (HF) is presented. By this procedure three critical steps are required to attain the bulk lifetime. Firstly, prior to immersing silicon wafers into HF, they are chemically cleaned and subsequently etched in 25% tetramethylammonium hydroxide. Secondly, the chemically treated wafers are then placed into a large plastic container filled with a mixture of HF and hydrochloric acid, and then centered over an inductive coil for photoconductance (PC) measurements. Thirdly, to inhibit surface recombination and measure the bulk lifetime, the wafers are illuminated at 0.2 suns for 1 min&#160;using a halogen lamp, the illumination is switched off, and a PC measurement is immediately taken. By this procedure, the characteristics of bulk silicon defects can be accurately determined. Furthermore, it is anticipated that a sensitive RT surface passivation technique will be imperative for examining bulk silicon defects when their concentration is low (&#60;1012 cm-3).

A RT liquid surface passivation technique to investigate the recombination activity of bulk silicon defects is described. For the technique to be successful, three critical steps are required: (i) chemical cleaning and etching of silicon, (ii) immersion of silicon in 15% hydrofluoric acid and (iii) illumination for 1 min.

A procedure to measure the bulk lifetime (&#62;100 &#181;sec) of silicon wafers by temporarily attaining a very high level of surface passivation when ...

Frequency Mixing Magnetic Detection Scanner for Imaging Magnetic Particles in Planar Samples

The setup of a planar Frequency Mixing Magnetic Detection (p-FMMD) scanner for performing Magnetic Particles Imaging (MPI) of flat samples is presented. It consists of two magnetic measurement heads on both sides of the sample mounted on the legs of a u-shaped support. The sample is locally exposed to a magnetic excitation field consisting of two distinct frequencies, a stronger component at about 77 kHz and a weaker field at 61 Hz. The nonlinear magnetization characteristics of superparamagnetic particles give rise to the generation of intermodulation products. A selected sum-frequency component of the high and low frequency magnetic field incident on the magnetically nonlinear particles is recorded by a demodulation electronics. In contrast to a conventional MPI scanner, p-FMMD does not require the application of a strong magnetic field to the whole sample because mixing of the two frequencies occurs locally. Thus, the lateral dimensions of the sample are just limited by the scanning range and the supports. However, the sample height determines the spatial resolution. In the current setup it is limited to 2 mm. As examples, we present two 20 mm &#215; 25 mm p-FMMD images acquired from samples with 1 &#181;m diameter maghemite particles in silanol matrix and with 50 nm magnetite particles in aminosilane matrix. The results show that the novel MPI scanner can be applied for analysis of thin biological samples and for medical diagnostic purposes.

A scanner for imaging magnetic particles in planar samples was developed using the planar frequency mixing magnetic detection technique. The magnetic intermodulation product response from the nonlinear nonhysteretic magnetization of the particles is recorded upon a two-frequency excitation. It can be used to take 2D images of thin biological samples.

The setup of a planar Frequency Mixing Magnetic Detection (p-FMMD) scanner for performing Magnetic Particles Imaging (MPI) of flat samples is presented. ...

Detecting the Water-soluble Chloride Distribution of Cement Paste in a High-precision Way

To improve the accuracy of the chloride distribution along the depth of cement paste under cyclic wet-dry conditions, a new method is proposed to obtain a high-precision chloride profile. Firstly, paste specimens are molded, cured, and exposed to cyclic wet-dry conditions. Then, powder samples at different specimen depths are grinded when the exposure age is reached. Finally, the water-soluble chloride content is detected using a silver nitrate titration method, and chloride profiles are plotted. The key to improving the accuracy of the chloride distribution along the depth is to exclude the error in the powderization, which is the most critical step for testing the distribution of chloride. Based on the above concept, the grinding method in this protocol can be used to grind powder samples automatically layer by layer from the surface inward, and it should be noted that a very thin grinding thickness (less than 0.5 mm) with a minimum error less than 0.04 mm can be obtained. The chloride profile obtained by this method better reflects the chloride distribution in specimens, which helps researchers to capture the distribution features that are often overlooked. Furthermore, this method can be applied to studies in the field of cement-based materials, which require high chloride distribution accuracy.

A protocol for obtaining a water-soluble chloride profile by using a high precision milling method is presented.

To improve the accuracy of the chloride distribution along the depth of cement paste under cyclic wet-dry conditions, a new method is proposed to obtain a ...

Evaluation of Commercial-Off-The-Shelf Wrist Wearables to Estimate Stress on Students

Wearable commercial-off-the-shelf (COTS) devices have become popular during the last years to monitor sports activities, primarily among young people. These devices include sensors to gather data on physiological signals such as heart rate, skin temperature or galvanic skin response. By applying data analytics techniques to these kinds of signals, it is possible to obtain estimations of higher-level aspects of human behavior. In the literature, there are several works describing the use of physiological data collected using clinical devices to obtain information on sleep patterns or stress. However, it is still an open question whether data captured using COTS wrist wearables is sufficient to characterize the learners' psychological state in educational settings. This paper discusses a protocol to evaluate stress estimation from data obtained using COTS wrist wearables. The protocol is carried out in two phases. The first stage consists of a controlled laboratory experiment, where a mobile app is used to induce different stress levels in a student by means of a relaxing video, a Stroop Color and Word test, a Paced Auditory Serial Addition test, and a hyperventilation test. The second phase is carried out in the classroom, where stress is analyzed while performing several academic activities, namely attending to theoretical lectures, doing exercises and other individual activities, and taking short tests and exams. In both cases, both quantitative data obtained from COTS wrist wearables and qualitative data gathered by means of questionnaires are considered. This protocol involves a simple and consistent method with a stress induction app and questionnaires, requiring a limited participation of support staff.

A protocol to evaluate solutions based on commercial-off-the-shelf (COTS) wrist wearables to estimate stress in students is proposed. The protocol is carried out in two phases, an initial laboratory-based stress induction test, and a monitoring stage taking place in the classroom while the student is performing academic activities.

Wearable commercial-off-the-shelf (COTS) devices have become popular during the last years to monitor sports activities, primarily among young people. ...

Radio Frequency Magnetron Sputtering of GdBa2Cu3O7&#8722;&#948;/ La0.67Sr0.33MnO3 Quasi-bilayer Films on SrTiO3 (STO) Single-crystal Substrates

Here, we demonstrate a method of coating ferromagnetic La0.67Sr0.33MnO3 (LSMO) nanoparticles on (001) SrTiO3 (STO) single-crystal substrates by radio frequency (RF) magnetron sputtering. LSMO nanoparticles were deposited with diameters from 10 to 20 nm and heights between 20 and 50 nm. At the same time, (Gd) Ba2Cu3O7&#8722;&#948; ((Gd) BCO) films were fabricated on both undecorated and LSMO nanoparticle decorated STO substrates using RF magnetron sputtering. This report also describes the properties of GdBa2Cu3O7&#8722;&#948;/ La0.67Sr0.33MnO3 quasi-bilayer films structures (e.g., crystalline phase, morphology, chemical composition); magnetization, magneto-transport, and superconducting transport properties were also evaluated.

Radio Frequency Magnetron Sputtering of GdBa2Cu3O7ÃƒÆ’Ã‚Â¢Ãƒâ€¹Ã¢â‚¬Â 'ÃƒÆ’Ã…Â½ Â´/ La0.67Sr0.33MnO3 Quasi-bilayer Films on SrTiO3 STO Single-crystal Substrates

Here, we present a protocol to grow LSMO nanoparticles and (Gd) BCO films on (001) SrTiO3 (STO) single-crystal substrates by radio frequency (RF)-sputtering.

Here, we demonstrate a method of coating ferromagnetic La0.67Sr0.33MnO3 (LSMO) nanoparticles on (001) SrTiO3 (STO) single-crystal substrates by radio ...

A Rapid Method for Modeling a Variable Cycle Engine

The variable cycle engines (VCE) that combine the advantages of turbofan and turbojet engines, are widely considered to be the next generation aircraft engines. However, developing VCE requires high costs. Thus, it is essential to build a mathematical model when developing an aircraft engine, which may avoid a large number of real tests and reduce the cost dramatically. Modeling is also crucial in control law development. In this article, based on a graphical simulation environment, a rapid method for modeling a double bypass variable cycle engine using object-oriented modeling technology and modular hierarchical architecture is described. Firstly, the mathematical model of each component is built based on the thermodynamic calculation. Then, a hierarchical engine model is built via the combination of each component mathematical model and the N-R solver module. Finally, the static and dynamic simulations are carried out in the model and the simulation results prove the effectiveness of the modeling method. The VCE model built through this method has the advantages of clear structure and real-time observation.

Here, we present a protocol to build a component-level mathematical model for a variable cycle engine.

The variable cycle engines (VCE) that combine the advantages of turbofan and turbojet engines, are widely considered to be the next generation aircraft ...

Robotic Sensing and Stimuli Provision for Guided Plant Growth

Robot systems are actively researched for manipulation of natural plants, typically restricted to agricultural automation activities such as harvest, irrigation, and mechanical weed control. Extending this research, we introduce here a novel methodology to manipulate the directional growth of plants via their natural mechanisms for signaling and hormone distribution. An effective methodology of robotic stimuli provision can open up possibilities for new experimentation with later developmental phases in plants, or for new biotechnology applications such as shaping plants for green walls. Interaction with plants presents several robotic challenges, including short-range sensing of small and variable plant organs, and the controlled actuation of plant responses that are impacted by the environment in addition to the provided stimuli. In order to steer plant growth, we develop a group of immobile robots with sensors to detect the proximity of growing tips, and with diodes to provide light stimuli that actuate phototropism. The robots are tested with the climbing common bean, Phaseolus vulgaris, in experiments having durations up to five weeks in a controlled environment. With robots sequentially emitting blue light-peak emission at wavelength 465 nm-plant growth is successfully steered through successive binary decisions along mechanical supports to reach target positions. Growth patterns are tested in a setup up to 180 cm in height, with plant stems grown up to roughly 250 cm in cumulative length over a period of approximately seven weeks. The robots coordinate themselves and operate fully autonomously. They detect approaching plant tips by infrared proximity sensors and communicate via radio to switch between blue light stimuli and dormant status, as required. Overall, the obtained results support the effectiveness of combining robot and plant experiment methodologies, for the study of potentially complex interactions between natural and engineered autonomous systems.

Distributed robot nodes provide sequences of blue light stimuli to steer the growth trajectories of climbing plants. By activating natural phototropism, the robots guide the plants through binary left-right decisions, growing them into predefined patterns that by contrast are not possible when the robots are dormant.

Robot systems are actively researched for manipulation of natural plants, typically restricted to agricultural automation activities such as harvest, ...

Manufacturing, Control, and Performance Evaluation of a Gecko-Inspired Soft Robot

This protocol presents a method for manufacturing, control, and evaluation of the performance of a soft robot that can climb inclined flat surfaces with slopes of up to 84&#176;. The manufacturing method is valid for the fast pneunet bending actuators in general and might, therefore, be interesting for newcomers to the field of actuator manufacturing. The control of the robot is achieved by means of a pneumatic control box that can provide arbitrary pressures and can be built by only using purchased components, a laser cutter, and a soldering iron. For the walking performance of the robot, the pressure-angle calibration plays a crucial role. Therefore, a semi-automated method for the pressure-angle calibration is presented. At high inclines (&#62; 70&#176;), the robot can no longer reliably fix itself to the walking plane. Therefore, the gait pattern is modified to ensure that the feet can be fixed on the walking plane.

This protocol provides a detailed list of steps to be performed for the manufacturing, control and evaluation of the climbing performance of a gecko-inspired soft robot.

This protocol presents a method for manufacturing, control, and evaluation of the performance of a soft robot that can climb inclined flat surfaces with ...

Electroantennography-based Bio-hybrid Odor-detecting Drone using Silkmoth Antennae for Odor Source Localization

Small drones with chemical or biosensor devices that can detect airborne odorant molecules have attracted considerable attention owing to their applicability in environmental and security monitoring and search-and-rescue operations. Small drones with commercial metal-oxide-semiconductor (MOX) gas sensors have been developed for odor source localization; however, their real-time-odor-detection performance has proven inadequate. However, biosensing technologies based on insect olfactory systems exhibit relatively high sensitivity, selectivity, and real-time response with respect to odorant molecules compared to commercial MOX gas sensors. In such devices, excised insect antennae function as portable odorant biosensor elements and have been found to deliver excellent sensing performance. This study presents experimental protocols for odorant-molecule detection in the air using a small autonomous bio-hybrid drone based on a mountable electroantennography (EAG) device incorporating silkmoth antennae.
We developed a mountable EAG device including sensing/processing parts with a Wi-Fi module. The device was equipped with a simple sensor enclosure to enhance the sensor directivity. Thus, odor source localization was conducted using the spiral-surge algorithm, which does not assume an upwind direction. The experimental bio-hybrid odor-detecting drone identified real-time odorant-concentration differences in a pseudo-open environment (outside a wind tunnel) and localized the source. The developed drone and associated system can serve as an efficient odorant molecule-detection tool and a suitable flight platform for developing odor source localization algorithms owing to its high programmability.

This study introduces experimental protocols for a bio-hybrid odor-detecting drone based on silkmoth antennae. The operation of an experimental electroantennogram device with silkmoth antennae is presented, in addition to the structure of a bio-hybrid drone designed for odor source localization using the spiral-surge algorithm.

Small drones with chemical or biosensor devices that can detect airborne odorant molecules have attracted considerable attention owing to their ...

Interactive and Visualized Online Experimentation System for Engineering Education and Research

Experimentation is crucial in engineering education. This work explores visualized experiments in online laboratories for teaching and learning and also research. Interactive and visualizing features, including theory-guided algorithm implementation, web-based algorithm design, customizable monitoring interface, and three-dimensional (3-D) virtual test rigs are discussed. To illustrate the features and functionalities of the proposed laboratories, three examples, including the first-order system exploration using a circuit-based system with electrical elements, web-based control algorithm design for virtual and remote experimentation, are provided. Using user-designed control algorithms, not only can simulations be conducted, but real-time experiments can also be conducted once the designed control algorithms have been compiled into executable control algorithms. The proposed online laboratory also provides a customizable monitoring interface, with which users can customize their user interface using provided widgets such as the textbox, chart, 3-D, and camera widget. Teachers can use the system for online demonstration in the classroom, students for after-class experimentation, and researchers to verify control strategies.

This work describes an online experimentation system that provides visualized experiments, including the visualization of theories, concepts, and formulas, visualizing the experimental process with three-dimensional (3-D) virtual test rigs, and visualizing the control and monitoring system using widgets such as charts and cameras.

Experimentation is crucial in engineering education. This work explores visualized experiments in online laboratories for teaching and learning and also ...