Mixed Reality for Education (MRE) Implementation and Results in Online Classes for Engineering

Summary

In this work, a mixed reality system called MRE was developed to help students develop laboratory practices complementing online classes. An experiment was carried out with 30 students; 10 students did not use MRE, 10 used MRE, and 10 more used MRE with teacher feedback.

Abstract

The COVID-19 pandemic has changed many industries, empowering some sectors and causing many others to disappear. The education sector is not exempt from major changes; in some countries or cities, classes were taught 100% online for at least 1 year. However, some university careers need laboratory practices to complement learning, especially in engineering areas, and having only theoretical lessons online could affect their knowledge. For this reason, in this work, a mixed reality system called mixed reality for education (MRE) was developed to help students develop laboratory practices to complement online classes. An experiment was carried out with 30 students; 10 students did not use MRE, 10 used MRE, and 10 more used MRE with teacher feedback. With this, one can see the advantages of mixed reality in the education sector. The results show that using MRE helps to improve knowledge in engineering subjects; the students obtained qualifications with grades 10% to 20% better than those who did not use it. Above all, the results show the importance of feedback when using virtual reality systems.

Introduction

Technology has always been present in the education sector; profound changes have occurred in the devices used to teach classes. However, face-to-face classes remain the preferred option for students and teachers. When the pandemic came, it changed all sectors, and education was no exception. In 2018, before the pandemic, only 35% of students who studied a degree reported having taken at least one class online; that is, 65% of students completed their studies in person¹. As of April 2020, by government order (Mexican), all public and private schools were prohibited from teaching face-to-face classes; for this reason, 100% of the students had to take distance classes. Universities were the first to act, using tools for video calling, preparing classes, homework management, etc. This makes sense, since people of university age (between 18 and 25 years old) are people who have been in contact with technology since birth.

Some classes can be fully adapted virtually; however, laboratory practices are complex to perform remotely, and students do not have the necessary material, which is often expensive. The impact that online classes have on the quality of knowledge is unclear, and some studies show that online courses generally yield worse student performance than in-person coursework². But one thing is certain, not carrying out laboratory practices that bring students closer to what they will experience in the industry will negatively affect their professional performance. Therefore, the importance of real-scale experiences becomes necessary in the current teaching of engineering^3,4,5. For these reasons, new technologies are being used to mitigate these problems. Among them are virtual reality (VR), augmented reality (AR), and mixed reality (MR). It is important to mention that VR is a technology that allows the creation of a totally immersive digital environment, whileAR overlays virtual objects in the real-world environment. On the other hand, MR does not just use virtual objects, but also anchors these objects to the real world, making it possible to interact with them. Thus, MR is a combination of VR and AR⁶. On the other hand, some organizations have also made efforts to develop remote laboratories, where real equipment exists but can be controlled remotely⁷.

The term MR dates to 1994; however, in the last 5 years, it has taken on special importance, thanks to large companies that have focused their efforts on developing environments, such as the Metaverse⁶. MR can be applied in different areas; two of the most common are training and education. Training has been one of the great drivers of MR; it is very expensive for a company to stop a production line to train new employees, or in dangerous environments, and it isn't easy to carry out training in the field. Education is not far behind; although face-to-face classes have changed very little, there are great efforts to incorporate MR into classes^8,9. For education, there are professional careers where it is necessary to carry out laboratory practices to have complete training. Many existing studies and research are in medicine, with VR, AR, and MR playing a key role. Multiple papers show how MR surpasses traditional teaching methods in surgical and medical subjects, where the practice is a clear advantage for developing students^{10,11,12,13,14}.

However, there is not the same amount of research on engineering issues. Normally in engineering careers, a student has theory classes complemented by practices. In this way, there are studies on MR and VR showing the benefits in engineering pedagogy¹². However, some of these studies focus on analyzing the complexity of the environment and the tools used^8,15. Tang et al. devised a study where students from different areas and with different knowledge used MR to improve their understanding of geometric analysis and creativity¹⁶. In a subsequent test, people who took their classes using MR finished faster, making it clear that MR positively affects learning¹⁶. Moreover, Halabi showed the use of VR tools in engineering education. Although it is not MR, it shows tools that can be used for teaching. It makes a real case study to show that it is possible to introduce VR in engineering classes¹⁷.

On the other hand, remote laboratories (RLs) are technological tools composed of software and hardware that allow students to remotely carry out their practices as if they were in a traditional laboratory. RLs are generally accessed through the internet, and are normally used when students are required to autonomously put into practice what they have learned as many times as they require¹⁸. However, with the arrival of COVID-19, its use has been to replace traditional laboratories and to be able to carry out practices during online classes¹⁸. As mentioned above, an RL needs a physical space (traditional laboratory) and elements that allow it to be controlled remotely. With the arrival of VR, laboratories have been modeled virtually, and through physical mechanisms, the elements of the laboratory can be controlled¹⁹. However, having an RL is very expensive, impeding many schools especially in developing countries. Some studies mention that costs can vary between $50,000 and $100,000^20,21.

Moreover, since the pandemic began, changes have had to be made quickly; in the case of RLs, attempts were made to send kits to the homes of each student to replace the traditional laboratories. However, there was a cost problem, as studies showed that each kit cost around $700^18,22. Nevertheless, the studies used expensive and difficult-to-obtain components. The pandemic affected education worldwide, and not many people could spend thousands of dollars to automate a lab or buy a kit. Most studies consider face-to-face classes and complement them with MR. However, in recent years, classes have been online due to COVID-19, and only some works show the improvement of virtual classes using MR and affordable devices^23,24.

The research that exists so far is mainly focused on medicine, with little information on engineering. However, without a doubt, we believe that the greatest contribution and difference is that our experiment was carried out for 6 months and was compared with subjects with the same characteristics who did not use virtual models, whereas most previous works carried out short experiments to compare single technologies or procedures; they did not apply them over several months. Therefore, this paper shows the difference in learning that can be made using MR in a university subject.

For this reason, this work shows the development and results of an MR system to help carry out laboratory practices in universities focused on electronic engineering. It is important to mention that special emphasis is placed on keeping the cost of the device low, making it accessible to the general population. Three groups use different teaching methods, and an exam is conducted on the class topics. In this way, it is possible to obtain results on understanding the topics in distance education using MR.

The project explained in this work is called mixed reality for education (MRE) and is proposed as a platform where students use VR glasses with a smartphone (i.e., no special VR glasses are used). A workspace is created where students can interact with virtual environments and real objects simply by using their own hands, due to the use of virtual and real objects, a mixed reality system. This workspace consists of a base with an image where all the virtual objects are displayed and interacted with. The environment created focuses on conducting laboratory practices to show electronic components and physics for engineering careers. It is important to highlight the need to provide feedback to students. For this reason, MRE incorporates a feedback system where an administrator (normally the teacher) can see what is being done to rate the activity. In this way, feedback can be given on the work done by the student. Finally, the scope of this work is to check if there are advantages in using MR in online classes.

To achieve this, the experiment was carried out with three groups of students. Each group consisted of 10 students (30 students in total). The first group did not use MRE, only taking theory (online classes) on the momentum conservation principle and electronic components. The second group used MRE without feedback, and the third group used MRE with feedback from a teacher. It is important to mention that all students have the same school level; they are university students in the same semester and with the same career, studying mechatronics engineering. The experiment was applied in a single course called Introduction to Physics and Electronics, in the second semester of the degree; that is, the students had been in university for less than 1 year. Therefore, the topics covered in the class can be considered basic from an engineering point of view. The experiment was carried out on 30 students, as this was the number of students who enrolled in the class where the experiment was authorized. The selected class (Introduction to Physics and Electronics) had theory and laboratory practices, but due to the pandemic, only theory classes were being taught. The students were separated into three groups to see the impact that the practices have on general learning and if MR classes could be a substitute for face-to-face practices.

Protocol

The protocol follows the guidelines of the Panamerican University ethics committee. The experiment was conducted with a total of 30 students, between 18 and 20 years old; eight students were female and 22 were male, and they all attended the Panamerican University in Guadalajara, Mexico (the second largest city in Mexico). All participants completed the informed consent process and provided written permission for photos to be taken and published during data collection. The only requirement was that the students needed to have a smart phone, which was no problem. Therefore, there was no exclusion criteria for the experiment.

1. VR system setup and calibration

NOTE: This step takes ~10 min.

1. Ensure the system includes all the components: an Android phone with operating system version 10 or higher, VR box glasses, and a wooden base with a calibration image (Figure 1) (see Table of Materials).
2. Open the MRE application on the cell phone and load the Unity, AR Foundation, Google Cardboard, and ManoMotion services^25,26,27,28. The MRE application has been developed by ourselves; it was developed for Android and it is not public.
3. Insert the cell phone into the VR glasses and put on the glasses.
4. Visually locate the center of the base of the MRE prototype (the blue square in Figure 1).
5. When the simulation appears, raise an outstretched hand to place it in the center of the view.
   NOTE: From this moment, the users can make hand gestures to interact with the simulated environment.

2. User preparation

NOTE: This step takes ~5 min.

1. Without VR glasses, open the MRE application, as shown in Figure 2.
2. Ensure that the application starts in user mode so it is only necessary to log in.
3. Select the scenario that the user wants to perform. There are two scenarios: electronic components and physics.
4. Press Play; the user will have 30 s to put on the VR glasses.

3. Execution of scenarios

NOTE: This step takes ~15 min.

1. Scenario 1: electronic components
   1. Locate the areas to position components, by means of red, green, and blue colors. This delimits the six interaction zones of this scene: three zones to take the virtual electronics components and three zones to drop the components, as shown in Figure 3.
   2. Take the component and position it in the right place. The right place depends on the component and what is seen in theory; for example, in theory, it is explained how to place a heat sink, and in MRE said placement is practiced.
   3. Continue until all components are in place.
2. Scenario 2: physics
   1. Locate the two cars involved in the scenario (Figure 4).
   2. Select the speed of each car.
   3. Visualize the graphs after the collision.

4. Administration view

1. On the main screen, press MRE modes (see Figure 2) and select the administrator option.
2. Log in to verify if the account has permission to access as admin.
   ​NOTE: It becomes possible to view the list of students and the grades obtained in each scenario.

5. Student results

1. Logging in as an administrator, click on the name of the desired student and view the table with the information of the grades of their scenarios.
2. Click on a student's name and select download grades as CSV. This will display all the results in a comma separated file.

Results

This section shows the results obtained from the experiment. First, some details of how the experiment was carried out are explained, then the tests carried out on the students of the experiment are shown, and moreover, the results of the tests are presented. Finally, an analysis using one student of each group is described.

One of the biggest problems that the pandemic brought to engineering education was that it was not possible to carry out face-to-face laboratory practices, which has a dir...

Discussion

The MRE system allows different scenarios for students to learn about electronic components or physics topics. An important point is the possibility of the teacher providing feedback. In this way, the students can know what they did wrong and why. With the MRE system developed, an experiment was carried out with 30 students, where 10 students did not use MRE, 10 used MRE, and finally another 10 used MRE and received feedback from the teacher. At the end of the classes, a general knowledge test was given to all the studen...

Materials

Name	Company	Catalog Number	Comments
MRE application for Andorid			The application was developed for the experiment, it was made by us. It is NOT public, and there are no plans for publication.
Non-slip fabric (20 x 20 cm)
Printing of our base image
Self-adhesive paper (1 letter size sheet)
Virtual Reality Glasses	Meta Quest 2		We use the Meta Quest 2, which is a virtual reality headset with two displays of 1832 x 1920 pixels per eye, with this headset you could play video games, or try simulators with a 360 view. Also, the headset has two controls, in which the virtual hands feel like your real ones and this is thanks to the hand-tracking technology. https://www.meta.com/quest/products/quest-2/tech-specs/#tech-specs
Wooden plate (20 x 20 cm)