This manuscript&#160;was supported by funding from the National Science Foundation (DRL#1660878, DRL#1661202, DUE#1761178, DRL#1916417), the Social Sciences and Humanities Research Council of Canada (SSHRC 895-2011-1006), the Ministry of Sciences and Innovation I+D+i (PID2019-107201GB-100), and the European Union through the European Regional Development Funds (ERDF) and the Principality of Asturias (FC-GRUPIN-IDI/2018/000199). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation or Social Sciences and Humanities Research Council of Canada.&#160;The authors would also like to thank members of the SMART Lab at UCF for their assistance and contributions.

The research ethics committee of the Principality of Asturias and the University of Oviedo approved this protocol.
1. Session 1: diagnosis assessment
NOTE: In this session of the protocol, evaluation tests from different publishers are used, which have their own specific application and interpretation manuals. Since these tests, or other similar ones, are widely known by the scientific community in the field of psychology and education, the procedure to apply them is ...

The authors have nothing to disclose.

The current protocol proposes a multimodal evaluation focused on metacognitive, self-regulation, and emotional processes, which make up the basis of the difficulties in adults with LDs.
Session 1 is essential because it is intended to be a diagnostic assessment of the participant&#8217;s learning disabilities. Note that this session here is carried out by therapists with experience in diagnosing developmental and learning difficulties in the research and clinical context. We use these tools in...

Specific learning disorders (SLDs) encompass disorders of those who have difficulty learning and using academic skills, exhibiting performance below expectations for their chronological age in the areas of reading, writing, and/or mathematics1,2. There are different estimations of prevalence rates depending on the age, language and culture analyzed but they are between 5% and 15%1,3. Within the global category of neurodevelopmental disorders in the Diagnostic and Statistical Manual of Mental Disorders (5th Ed.)1, it is also necessary to focus on the incidence of Attention-Deficit/Hyperactivity Disorder (hereinafter ADHD) as it is a common disorder that has given rise to various controversies about how to approach it in recent years. Based on the DSM-51, it can be defined as a pattern of persistent behaviors of inattention and/or hyperactivity-impulsivity. Likewise, autism spectrum disorder (hereinafter ASD) is a category in the same manual that includes students who present neurodevelopmental disorders as a result of multifactorial dysfunctions of the central nervous system, which result in qualitative dysfunctions in three fundamental areas of the development of the person: social interaction, communication and interests and behaviors1,2.
On these lines, a new concept has emerged moving away from the sense of deficit and offering a more positive approach to these disorders to be consistent with current ideas of neurodevelopmental difficulties as highly coexistent and overlapping4. From these new models, it is understood that the skills involved in high-level cognitive processes, which allow managing and regulating one&#39;s behavior in order to achieve a desired goal, are crucial for self-regulation and, therefore, for activities of daily living, including the academic ones5. In the context of adulthood, neurodiversity has evolved to include various types of difficulties, including ADHD and ASD, as well as dyslexia, dyspraxia, and/or dyscalculia. Accordingly, we are approaching this neurodiversity from a broad conception of learning difficulties (LDs). The increase in students with this diversity enrolled in postsecondary education is well documented and is due, in part, to the increase in high school graduation rates for students with disabilities6, but at the same time, there is less research about the learning process of these students than necessary7.
Each of the disorders approached in isolation involve different deficits and manifestations; however, some commonality can be found within that heterogeneity in terms of LD, such as metacognitive, self-regulatory, and emotional malfunctioning8,9,10,11. Three fundamental foundations in the literature of learning in general, and LDs in particular, that represent the basis of successful learning and play an essential role in these well-known difficulties at the academic level12. As well as this, other approaches understand that there could be a certain commonality between deficits in executive functions, such as problems in automatic processing or working memory, that occur in different disorders such as ADHD and reading disorders13 or ADHD and ASD5. However, there is still work to be done in this field, since not all studies reach the same conclusions about these points in common in relation to executive functions. It could be due to the variations presented by the samples from which the studies are based and the evaluation procedures of the executive functions used in the investigations5,14.
In educational terms, this diverse mix affects not only the quality of learning, due to the fundamental nature of the affected functions, but also phenomena such as school dropout, change of degree, etc., with economic implications for governments and universities15. The dropout rate for students with LDs is higher than for students in the general population16 but also higher than the dropout rates for any other category of psychological disabilities except for those students with emotional disturbances17. In contrast, the number of students with LDs who are accessing post-compulsory education (vocational training, college, etc.) is increasing15, specifically in higher education19,20,21,22. Moreover, one might well assume that there are many more students with LD than those who officially pass through student services and typically make up the prevalence statistics23.
These difficulties are not always detected during childhood, especially in adults born before these disorders were considered in the regular academic system, and the symptoms of these disorders persist throughout people&#8217;s lives and cause difficulties in work, education and personal lives24. Research has shown that although people might overcome some of their difficulties, most continue to exhibit struggles with learning during adulthood and their persistence is still problematic at those higher educational levels25.
Paradoxically, unlike in previous educational levels and earlier ages, there are hardly any evidence-based instruments or evaluation protocols for adults with LDs. Despite the proliferation of diagnostic tools to evaluate LDs during childhood, the availability of valid, reliable instruments and methodologies for the adult population is significantly limited24. A recent literature review about learning disabilities in higher education found that most of the information collected in this regard is done through interviews, and only occasionally are self-report questionnaires used26. Self-report methodology and interviews, although valuable, are not enough to accurately assess metacognitive, self-regulation, and emotional skills processes, in fact, among others, because of the process nature. The importance of scales and interview methodology for measuring those processes is undeniable27,28, but so too are the associated problems of validity29 and incongruence with other innovative methods of assessment30. An additional problem in the detection of LDs is the bias in the diagnosis of the disorder due to the absence of comprehensive assessment protocols. The fact that professionals do not have a reference protocol based on objective variables is frequently causing many false positive and false negative cases of LDs31.
In response to both scarcity of instruments for adults and the need to improve existing methodology, the current study proposes a multimodal evaluation protocol focused on metacognitive, self-regulation, and emotional processes, which make up the basis of the difficulties in adults with LDs. In line with the current literature, we propose a move toward integrative and multichannel measurement32,33. The assessment is carried out through an analysis of the on-line learning process using several methods, techniques, and sensors (e.g., hypermedia learning environment, virtual reality, eye tracking, facial expressions of emotion, physiological responses, log files, screen recordings of human-machine interactions) and off-line methods (e.g., questionnaires, interviews, and self-report measures). This mixed methodology provides evidence of the deployment of target processes before, during, and after learning that can be triangulated to enhance the understanding of how students learn and where the problem lies, if there is one34.
The evaluation protocol is carried out over two sessions. The sessions can be done in one sitting or may need partial applications depending on the person. The first is focused on the detection or confirmation of LDs and what specific kind of disorder we are facing, and the second is designed to go into the metacognitive, self-regulation, and emotional processes of each individual case in depth.
Session 1 is intended to be a diagnostic or confirmation assessment of the participant&#8217;s learning disabilities: SLD, ADHD and/or ASD (high functioning) to determine what type of specific problems the participants have. This assessment is essential for two reasons. 1) Adults with Learning Disabilities rarely have accurate information about their dysfunctional behavior. Some of them suspect that they have a LD but have never been evaluated. Others may have been assessed when they were children but do not have any reports or further information. 2) There may be discrepancies with previous diagnoses (e.g., a previous dyslexia diagnosis as opposed to a current diagnosis of attention deficit and slow processing speed; previous ASD diagnosis in contrast to current limited intellectual ability, etc.). The participant is interviewed, and questionnaires and standardized tests are applied. This session here is carried out by therapists with experience in diagnosing developmental and learning difficulties in the research and clinical context in different offices of a Spanish Psychology Faculty. The session begins with a structured interview that collects biographical information along with the presence of symptoms related to SLDs that are referred to in the DSM-51. Following that, the reference intellectual ability test WAIS-IV35 is used in case of exclusion criterion implementation and because it provides very valuable information for learning difficulties from the scales &#8220;work memory&#8221; and &#8220;processing speed&#8221;36. Additionally, the PROLEC SE-Revised Test37 is extensively used to evaluate reading disabilities (lexical, semantic and/or syntactic processes of reading), one of the most prevalent and disabling difficulties for learning in current academic contexts, which overlaps with other disorders such as ADHD38. This evaluation collects reading accuracy, speed and fluency along with reading disabilities, and more importantly, in which reading process the failure occurs37 (this test has been evaluated with pre-university students. Currently, there are no tests in Spain that are adapted to the general adult population, so this test was selected because it is the closest to the target population). Then, we screen symptoms of ADHD through the World Health Organization Adult ADHD Self-Report Scale (ASRS)39 and refine the evaluation of this disorder, introducing multimodality with a cutting-edge virtual reality continuous performance test for the evaluation of attentional processes and working memory in adults, the Nesplora Aquarium31,40. This test is a very useful tool when diagnosing ADHD in adults and adolescents over 16 years old in an ecological scenario, providing objective, reliable data. It evaluates selective and sustained attention, impulsivity, reaction time, auditory and visual attention, perseverance, quality of attentional focus, motor activity, work memory and cost of change of task. Additionally, along with the WAIS-IV35 as a whole for collecting information about the participant&#8217;s intellectual ability, we pay special attention to the scales &#8220;work memory&#8221; and &#8220;processing speed&#8221; because they are related to learning difficulties and the results of these scales are used in the final decision. Finally, we include the Autism Spectrum Quotient (AQ-Short)41 in the protocol, the short version of the reliable AQ-Adult from Baron-Cohen, Wheelwright, Skinner, Martin and Clubley42.
Session 2 focuses on a multimodal assessment of the participant&#8217;s learning process. The key to understanding complex learning lies in understanding the deployment of students&#8217; cognitive, metacognitive, motivational, and affective processes43. To that end, participants work with MetaTutor, where the use of metacognitive and cognitive strategies deployed are observed while they are learning. MetaTutor is a hypermedia learning environment that is designed to detect, model, trace, and foster students&#8217; self-regulated learning while learning different science topic44. The design of MetaTutor is based on extensive research by Azevedo and colleagues43,45,46,47 and belongs to a new trend in the measurement of SRL, the so called third wave, which is characterized by combined use of measurement and advanced learning technologies33. The use of MetaTutor also provides multimodal trace data, incorporating measures such as, eye tracking, emotional physiological responses (galvanic skin response (GSR) and facial expressions of emotions)48, log-data and questionnaires. All these measures are combined to reach a deeper understanding of the participants SRL and metacognition.
Eye tracking provides an understanding of what attracts immediate attention, which target elements are ignored, in which order elements are noticed, or how elements compare to others; electrodermal activity lets us know how emotional arousal changes in response to the environment; facial-emotion-recognition allows the automatic recognition and analysis of facial expressions; and data logging collects and stores the student&#180;s interaction with the learning environment for further analysis. Concerning the questionnaires, the Mini International Personality Item Pool49 informs about a range of activities and thoughts that people experience in everyday life assessing each of the five major personality traits (extraversion, agreeableness, conscientiousness, neuroticism and openness). The Connotative Aspects of Epistemological Beliefs50 provides information about participants&#8217; beliefs about knowledge. The Rosenberg Self-esteem scale shows how the participants feel about themselves overall51. The Emotion Regulation Questionnaire52 provides information about participants&#8217; emotion regulation. The Achievement Emotions Questionnaire (AEQ)53 informs about emotions typically experienced at university.
In short, assessing LDs during adulthood is particularly difficult. Education and experience allow many adults to compensate for their deficits and later show undifferentiated or masked symptoms, on which scientific knowledge is still scarce. Taking into account the critical research gap that arises, this current work aims to ensure theoretically-driven, empirically-based guidelines for accurate assessment of LDs during adulthood in order to design effective prevention and intervention actions.
To help readers decide whether the method described is appropriate or not, it is necessary to specify that the protocol is not suitable for people with intellectual disabilities because their diagnosis invalidates the diagnosis of learning difficulties. In addition, due to the singularities of the equipment used and the format of showing the learning content, it is still not possible to evaluate people with motor disabilities (upper limbs, neck and/or face), hearing or visual impairment. Nor would it be suitable for participants with severe psychiatric disorders. It would require the use of drugs that could alter information processing or the physiological expression of emotions.

<table border="0" cellpadding="0" cellspacing="0" style="width:787px;" width="787">
	<colgroup>
		<col />
		<col />
		<col />
		<col />
	</colgroup>
	<tbody>
		<tr height="21">
			<td height="21" style="height:21px;width:216px;">AQUARIUM</td>
			<td style="width:242px;">Nesplora</td>
			<td style="width:161px;"></td>
			<td style="width:167px;"></td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:216px;">Eye-tracker RED500 Systems</td>
			<td style="width:242px;">SensoMotoric Instruments GmbH</td>
			<td style="width:161px;"></td>
			<td></td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:216px;">Face API</td>
			<td style="width:242px;">Microsoft</td>
			<td style="width:161px;"></td>
			<td></td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:216px;">GSR NUL-217</td>
			<td style="width:242px;">NeuLog</td>
			<td style="width:161px;"></td>
			<td></td>
		</tr>
	</tbody>
</table>

multimodal protocol for assessing metacognition and self-regulation in adults with learning difficulties

Learning disabilities (LDs) encompass disorders of those who have difficulty learning and using academic skills, exhibiting performance below expectations for their chronological age in the areas of reading, writing, and/or mathematics. Each of the disorders making up the LDs involve different deficits; however, some commonalities can be found within that heterogeneity, such in terms of learning self-regulation and metacognition. Unlike in early ages and later educational levels, there are hardly any evidence-based evaluation protocols for adults with LDs. LDs influence academic performance but also have serious consequences in professional, social, and family contexts. In response to this, the current work proposes a multimodal evaluation protocol focused on metacognitive, self-regulation of learning, and emotional processes, which make up the basis of the difficulties in adults with LDs. The assessment is carried out through analysis of the on-line learning process using a variety methods, techniques, and sensors (e.g., eye tracking, facial expressions of emotion, physiological responses, concurrent verbalizations, log files, screen recordings of human-machine interactions) and off-line methods (e.g., questionnaires, interviews, and self-report measures). This theoretically-driven and empirically-based guideline aims to provide an accurate assessment of LDs in adulthood in order to design effective prevention and intervention proposals.

This section illustrates the representative results obtained from the protocol, including an example of conjoint results of Session 1 and an example of each source of information from Session 2.
The results about disorders are collected in Session 1 through diagnostic tests taking into account the procedures and cut-off points specified for the diagnostic assessment of participants&#8217; learning difficulties (SLD, ADHD, and ASD). The expert committee decides whether the participant has learn...

Watch this Scientific Journal Video about Multimodal Protocol for Assessing Metacognition and Self-Regulation in Adults with Learning Difficulties at JoVE.com

Multimodal Protocol for Assessing Metacognition and Self-Regulation in Adults with Learning Difficulties

The current work proposes a multimodal evaluation protocol focused on metacognitive, self-regulation of learning, and emotional processes, which make up the basis of the difficulties in adults with LDs.

Learning disabilities (LDs) encompass disorders of those who have difficulty learning and using academic skills, exhibiting performance below expectations ...

Multimodal protocol for assessing metacognition and self-regulation in adults with learning difficulties. Specific learning disabilities encompass different disorders of those who have difficulties in learning and applying academic skills. Showing a performance below expectations in different areas, like reading, writing, or mathematics.Each disorder implies different deficits but we can find some commonalities on that like metacognitive, self regulation, and emotional malfunctioning. However, there are hardly any evaluation protocol for adults with learning disabilities. In response to this, we want to propose a Multimodal Assessment Protocol focusing on self regulation, metacognition, and emotional processes happening during learning.The evaluation is carried out through both online and offline methodology. We use different techniques, among others, eye tracking, face emotion recognition, galvanic skin response, log data analyze, and of course, interviews, questionnaires and self-reports. And soon we want to provide to the research community with some theoretically driving and empirically based guidelines for an accurate assessment of adults with learning disability in order to promote efficient intervention and prevention actions.Informed consent. Explain to the participants the ethical and confidentiality aspect of the research. And lead them to acknowledge and sign the individual informed consent.Structured interview. Explain to the participants how the session will be performed, and following the script, collect the biographical information along with the presence of symptoms related to a specific learning disabilities that are referred in the DSM V.First decision point. Finish the assessment if the participant meets the initial exclusion criteria There is, if they show a moderate disability in upper segments, a sensory disability, visual, or auditory, a diagnosis of intellectual disability and/or a serious mental disorder.Continue the assessment if it seems that the participant has a specific learning disability and no meet exclusion criteria. Intellectual ability. Apply the Wechsler Adult Intelligence Scale, fourth edition for collecting information about participant's intellectual ability following the instructions of the manual.Second decision point. Finish the assessment if the participant doesn't understand the instructions of the test or it cannot be evaluated or have an intelligence quotient less than 70. Attention Deficit Hyperactivity Disorder.Ask the participant to complete the self-reported screening questionnaire for all those of the World Health Organization in order to collect information about the presence of symptoms related to ADHD disorder. If the participant scores 12 or more in this questionnaire, apply the test to the subject. Reading difficulties.Apply the Revised screening test for reading difficulties following the instructions of the manual. Autism Spectrum Disorder. Ask the participant to complete the Autism Spectrum Quotient Questionnaire which provides information on the presence of symptoms related to social behavior, social skills, routines, switching, imagination, and numbers patterns.Last step of the session one. Analyze the results. Analyze each participant's interview questionnaires and test results and decide if they have significant learning difficulties or not.Two members of the experts committee, the evaluator and other member of the research team, analyze each participant's learning profile and decide if he or she is a student with a specific learning disabilities ADHD and/or Autism Spectrum Disorder or not. No tests can replace the expert's judgment. If the participant doesn't have any symptom the protocol is finished.If the participant show learning difficulties go to session two. Prepare the participant. Ask the participants tie back the hair, clear the neck, remove their glasses, and take chewing gum out if applicable.Clean the GSR and the participant's fingers with alcohol for skin. Galvanic Skin Response preparation and calibration. With the finger wristbands sensors on the index and ring fingers, with the connectors on the finger-tip side.Ask the participant to leave his hand on the table quietly and try to relax. Open the software in the computer. Make sure that the registration graph is working so the GSR is recording.Select the following sequence on the software. Run experiment, rate 10 per second, duration 10 minutes, and record. Now it's ready to record for 10 minutes to establish the baseline.Eye tracking and webcam preparation and calibration. Open the software in the computer, check that the two computers are connected to each other and that the eye tracking infrared lights are on and ready to capture the movement of the eyes. Adjust the webcam on the computer to the subject's position and ask the participant to sit facing forward and be as natural as possible.Click record. Write the registration data of the participant and then, press okay to start the calibration process. Ask the participant to press the space bar and follow the points on the screen with their eyes.Make sure that the participant's eyes, while looking at the screen, are centered before moving into the next step. The participant's gaze is centered when the movement of their eyes are registered on the side laptop screen with two white circles. Multimodal tracking of the learning session.Questionnaires and learning session in MetaTutor. Open the software and introduce the registration data of the participant. Every look will be recorded during the session in a data log file.Ask the participant about demographic and academic information. Before clicking continue, explain to the participants that they must follow the instructions that the tool will give them. Also, that they will be only interacting with the computer during the learning session.Ask the participant to answer the questionnaires about personality, self regulation, epistemological beliefs, and previous knowledge about the circulatory system. Show the participant the interface of MetaTutor and its different parts. The content area is where the learning content is displayed throughout the session in text form.They can navigate through a table of contents at the side of the screen to go to different pages. The overall learning goal is displayed at the top of the screen during the session. The sub goals line is set and displayed at the top in the middle of the screen.And they can manage sub goals or prioritize them here. A timer located at the top left corner of the screen displays the amount of time remaining in the session. As well as this, a list of self regulating processors are displayed in a pallet on the right hand side of the screen, and the participant can click on them for all the session to the deploy planning, monitoring, or learning strategies.Static images relevant to content pages are displayed beside the text to help learners coordinate the information from different sources. Text entered on the keyboard and the student's interactions with aliens are displayed and recorded at this part of the interface. Four artificial aliens help students in their learning throughout this session.These aliens are;Gavin the guy, Pam the planner, Mary the monitor, and Sam the strategizer. Ask the participant to click start the learning session whenever he's ready. The participant interacts with the tool.Once the session is finished ask the participant to complete the questionnaires again.Logoff. At the end of the session, save the recorded data of GSR eye tracking, webcam, and MetaTutor with the registration number of the participant. Extract the data as CSV files for easier use.Analysis of learning difficulties. Analyze each participant's learning performance based on the different reports obtained to result in a Multimodal profile. Remember, no report can replace the expert's judgment.This section illustrates some representative examples of results that can be obtained from the measures of session two. Firstly, we obtain a measure of GSR as an indication of emotional arousal during the learning session. Abrogates needs individual analysis by the expert committee taking into account each participant's specific baseline.Secondly, we obtain the predominant emotions during the learning process thanks to the Facial Emotion Recognition software. The results indicate the degree of coincidence with the analyzed emotions assigning values between zero and one to each of them. Thirdly, we extract eye tracking gaze information in terms of proportion of fixation time on pattern of fixations.For that purpose, we define seven areas of interest for self relation assessment in the MetaTutor interface. For instance, the image A shows a participant spending too much time in text content area, which could be indicating malfunction in regulation. Conversely, image B shows a participant who uses the learning tool resources in a balanced way.Fourthly, questionnaires are scored, according to the author's instructions. In this image, for example, we can observe the contours between the result obtained by different participants in self-esteem and even emotional regulation. Finally, all the learner's interactions with content, aliens and the learning environment, are recorded in logs for further detailed analysis.Previous experience allowed many adults to compensate their deficits and show individual characteristics on testing. Therefore, it is difficult to provide with accurate graph points for some of the data sources like GSR or log data aside in the target population. So result, the judgment of the experts to interpret the results as a whole, is still not replaceable.

multimodal-protocol-for-assessing-metacognition-self-regulation

Research

Education

JoVE Core

JoVE Journal

Introduction to Psychology

Behavior

Unit 1

Learning

SCIENTISTS IN ACTION

8.1K Views.  University of Oviedo. Multimodal protocol for assessing metacognition and self-regulation in adults with learning difficulties. Specific learning disabilities encompass different disorders of those who have difficulties in learning and applying academic skills. Showing a performance below expectations in different areas, like reading, writing, or mathematics.Each disorder implies different deficits but we can find some commonalities on that like metacognitive, self regulation, and emotional malfunctioning. ...