Our research focuses on exploring the feasibility of using artificial intelligence to measure and detect the student's level of attention in a classroom setting. We are investigating whether custom-developed algorithms and methodologies can effectively assess a student engagement during classes. The most recent approaches are based on artificial intelligence.
Specifically, the learning algorithms are a state of art. This kind of algorithm uses vast amount of data to train the linear models. And then they can be used to classify images, estimate the pose of a person, and perform a range of other interesting tasks.
We have two main critical challenges in our study. First, there are technical matters that must be solved. For instance, how to set up the devices, how to estimate the motion from the picture, the pose and the rest of other parameters we consider in our approach.
And then there is a subjective branch as it is unclear whether all individuals express the level of attention the same way. The strongest point of our system is that we use a multimodal approach. We analyze different inputs, facial emotion, body pose, head pose, accelerometer data, and heart rate to automatically classify the level of attention.
In addition, our methods heavily relies on artificial intelligence, which has reported been used with high success in other related research areas. To begin, set a router with wifi capabilities in the desired location to cover the entire room. Set one smartwatch and one camera for each student's location.
Set one embedded device for every two students. Fix two cameras on two tripods and connect them to another embedded device. Now connect the cameras to the corresponding embedded devices with a USB link and turn them on.
Turn on the smartwatches as well. Then connect every embedded device and smartwatch to the wifi network of the router. Place the cameras at the corner of each student's desk pointing forward and slightly tilted upwards to ensure the student's face is clearly visible.
Then, place the two tripods with cameras in front of the desks closest to the aisles in the first row of the classroom. Adjust the tripod riser to the highest position for a clear view of most students. Run the capturing software on smart watches and embedded devices to send images and accelerometer, gyroscope, heart rate and lighting data.
Additionally, run the server software to collect and store this data. Next, position the zenithal cameras to dominate the scene, ensuring a clear view of the student's bodies. Also place an additional camera in front of each student.
Ask the students to take their seats, inform them about the experiments goals, and instruct them to wear the smart watches on their dominant hand. Start the data capture and collection on the server and resume normal class lessons. To begin, set up the hardware and software required to detect attention levels in students.
Instruct the students to wear the smart watches on their dominant hand and start the data collection. Collect the required data by gathering all images and biometric data from each student's smartwatch. Then construct a data frame that includes data from one second comprising one image each from the individual and zenithal camera, 50 registers for each gyroscope, accelerometer, heart rate and light condition value.
Now use the estimated positions of the eyes, nose, mouth, and chin, provided by the algorithm to solve the perspective endpoint problem using the cv:solvePnP method of the openCV library. Obtain the rotation matrix that defines the head direction as a result. To compute the body pose, process the zenithal image using the BlazePose 18 landmark estimation algorithm and retrieve a list of 2D coordinates of the body's joints depicted in the image.
Obtain the student's face image and retrieve the eyes position by projecting their coordinates onto the image plane from the key points list. Now, trace a virtual vector joining both eyes'positions. Calculate the angle between this vector and horizontal line and construct a homography matrix.
Apply the reverse angle to the image and the key points to align the eyes horizontally. Then crop the face using the key points to identify its limits and feed the patch to the emotion detection convolutional neural network. Retrieve the network's output, a vector with seven positions representing probabilities for each of the seven basic emotions.
To measure the level of attention, obtain the accelerometer, gyroscope, and heart rate data from the data frame. After feeding the stream of accelerometer and gyroscope data to the custom built deep learning based model, retrieve the output from the model indicating the probabilities for handwriting, typing on a keyboard, using a mobile phone, or resting. Feed the final attention classifier by merging head direction, emotion recognition output, body pose, gyroscope, accelerometer, and heart rate data.
Then retrieve the classifier results and classify the continuous score into low, medium, and high attention categories. Show the results of the attention levels to the teacher using the graphical user interface from the teacher's computer, accessible through a regular web browser. The model was trained for 300 epochs.
The validation loss was less than 0.1%and the validation accuracy was 97%highlighting the good performance of the model.
