This work was supported by a project of the Principality of Asturias (FC-GRUPIN-IDI/2018/000199) and a predoctoral grant from the Ministry of Education, Culture, and Sports of Spain (FPU16/05802). We would like to thank Stephanie Jun for her help editing the English in the learning materials.

This protocol follows the Helsinki Declaration of Ethical Principles for Research with Humans, but applies these principles to the added difficulties of integrating research within real-life settings in education32. Specifically, neither the assignment of learning conditions nor the decision to participate can have consequences for students&#39; learning opportunities. In addition, confidentiality and the anonymity of students is maintained even when it is the teachers who are in charge of the eva...

The aim of this protocol is to guide researchers and educators in the implementation and evaluation of the PS-I approach in real classroom contexts. According to some previous experiences, PS-I can help promote deep learning and motivation in students19,21,24, but there is a need for more research about its efficacy in students with different abilities and motivational predispositions14,<sup cl...

One of the questions that teachers are most concerned about currently is how to stimulate students&#39; reflection. This concern is common in courses of a mathematical nature, such as STEM courses (Science, Technology, Engineering and Mathematics), in which the abstraction of many concepts requires a high degree of reflection, yet many students report approaching these courses purely through memory-based methods1. In addition, students often show superficial learning of the concepts1,2,3. The difficulties that students experience applying reflection and deep learning processes, however, are not only cognitive. Many students feel anxiety and demotivation faced with these courses4,5. In fact, these difficulties tend to persist throughout students&#39; educations6. It is therefore important to explore educational strategies that motivationally and cognitively prepare students for deep learning, regardless of their differing predispositions.
It is particularly useful to find strategies that complement typical instructional approaches. One of the most typical being direct instruction. Direct instruction means fully guiding students from the introduction of novel concepts with explicit information about these concepts, then following that with consolidation strategies such as problem-solving activities, feedback, discussions, or further explanations7,8. Direct instruction can be effective for easily transmitting content8,9,10. However, students often do not reflect on important aspects, such as how the content relates to their personal knowledge, or potential procedures that could work and do not11. It is therefore important to introduce complementary strategies to make students think critically.
One such strategy is the Problem-Solving before Instruction (PS-I) approach12, also referred to as the Invention approach11 or the Productive Failure approach13. PS-I is different to direct instruction in the sense that students are not directly introduced to the concepts, instead there is a problem-solving phase prior to the typical direct instruction activities in which students seek individual solutions to problems before getting any explanation about procedures for solving them.
In this initial problem, students are not expected to fully discover the target concepts13. Students may also feel cognitive overload14,15,16 and even negative affect17 with the uncertainty and the many aspects to consider. However, this experience can be productive in the long term because it can facilitate critical thinking about important features. Specifically, the initial problem can help students to become more aware of the gaps in their knowledge18,activate prior knowledge related to the content to cover13, and increase motivation because of the opportunity to base their learning on personal knowledge7,17,19.
In terms of learning, the effects of PS-I are generally seen when the results are evaluated with deep learning indicators20,21. In general no differences have been found between students who learned through PS-I and those who learned through direct instruction in terms of procedural knowledge20,22, which refers to the ability to reproduce learned procedures. However, students who go through PS-I generally exhibit higher learning in conceptual knowledge7,19,23, which refers to understanding the content covered, and transfer7,15,19,24, which refers to capacity to apply this understanding to novel situations. For example, a recent study in a class about statistical variability showed that students who were given the opportunity to invent their own solutions to measure statistical variability before receiving explanations about the general concepts and procedures in this topic demostrated better understanding at the end of the class than those who were able to directly study the relevant concepts and procedures before getting involved in any problem-solving activity23. However, some studies have shown no differences in learning16,25,26 or motivation19,26 between PS-I and direct instruction alternatives, or even better learning in direct instruction alternatives14,26, and it is important to consider potential sources of variability.
The design features underlying the implementation of PS-I are an important feature20. A systematic review20 found that there was more likely to be a learning advantage for PS-I over direct instruction alternatives when the PS-I interventions were implemented with at least one of two strategies, either formulating the initial problem with contrasting cases, or building the subsequent instruction with detailed feedback about the students&#39; solutions. Contrasting cases consist of simplified examples that differ in a few important characteristics11 (see Figure 1 for an example), and can help students identify relevant features and evaluate their own solutions during the initial problem11,20. The second strategy, providing explanations that build on the students&#39; solutions13, consist of explaining the canonical concept while giving feedback about the affordances and limitations of solutions generated by students, which can also help students focus on relevant features and evaluate the gaps in their own knowledge20, but after the initial problem-solving phase is completed (see Figure 3 for an example of the scaffolding from students&#39; typical solutions).
Given the support in the literature for these two strategies, contrasting cases and building instruction on students&#39; solutions, it is important consider them when promoting the inclusion of PS-I in real educational practice. This is the first goal of our protocol. The protocol provides materials for a PS-I intervention that incorporate these two principles. It is a protocol that,&#160;while adaptable, it is contextualized for a lesson on statistical variability, a very common lesson for university and high school students, who are generally the target populations in the literature on PS-I29. The initial problem-solving phase consists of inventing variability measures for income distributions in countries, which is a controversial topic30 that may be familiar to students in many learning areas. Then materials are provided for students to study solutions to this problem in a worked example, and for a lecture that incorporates discussion of common solutions produced by students along with embedded practice problems.
The second goal of our protocol is to make the experimental evaluation of PS-I accessible to educators and researchers, which can facilitate the investigation of PS-I from a greater variety of perspectives while maintaining some conditions constant across the literature. Yet conditions of this experimental evaluation are flexible to modifications.&#160;The experimental evaluation described in the protocol can be applied in ordinary lessons, since students in a single class can be assigned the materials for the PS-I condition or the materials for a direct instruction condition at the same time (Figure 4). This direct instruction condition is also adaptable to research and education needs, but as originally described in the protocol students start by getting the initial explanations about the target concept with the worked example, and then consolidate this knowledge with a practice problem (only presented in this condition to compensate for the time PS-I students spend on the initial problem), and with the lecture23. Potential adaptations include starting with the lecture and then having students to do the problem-solving activity, which is a typical control condition for comparing PS-I that has often led to better learning for the PS-I condition7,13,19,26. Alternatively, the control condition can be reduced to the exploration of a worked example followed by the lecture phase, which, although a more simplified version of direct instruction approaches than originally proposed, is more common in the literature and has led to varied results, with some studies indicating better learning in PS-I15,24, and others indicating better learning from this type of direct instruction condition14,26.
Finally, a third goal of the protocol is to provide resources for evaluating how students with different predispositions and cognitive abilities can benefit from PS-I15. The evaluation of these predispositions is especially important if we consider the negative predispositions that some students often have with STEM courses, and the fact that PS-I can still produce negative reactions in some cases14. There is, however, little research on this.
On the one hand, since PS-I facilitates the association of learning with individual ideas, rather than just formal knowledge, PS-I can be hypothesized as being able to help motivate students from low academic levels, those who have low feelings of competence, or low motivation about the subject13,27. One study showed that students with low mastery orientation, i.e., fewer goals related to personal learning, benefited more from PS-I than those with higher motivation to learn27. On the other hand, students with other profiles might encounter difficulties when involved in PS-I. More specifically, metacognition plays an important role in PS-I31, and students with low metacognition skills might not benefit from PS-I due to difficulties in being aware of their knowledge gaps or discerning relevant content15. In addition, as the initial phase of PS-I is based on the production of individual solutions, students with low divergent abilities, difficulties generating a variety of responses in a given situation, might benefit less from PS-I than other students. The protocol presents reliable instruments to assess for these predispositions (Table 1) although others may be considered.
In summary, this protocol aims to make an implementation of a PS-I intervention that follows accepted principles in the PS-I literature accessible to educators and researchers. Additionally, the protocols provide an experimental evaluation of this intervention, and facilitate the evaluation of students&#39; cognitive and motivational predispositions. It is a protocol that does not require access to new technologies or specific resources, and one that can be modified based on research and educational needs.

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>SPSS Program</td>
			<td>International Business Machines Corporation (IBM)</td>
			<td></td>
			<td>Other programs for general data analysis might be used instead</td>
		</tr>
		<tr>
			<td>PROCESS program</td>
			<td>Andrew F. Hayes (Ohio State University)</td>
			<td></td>
			<td>Freely accesible at: http://www.processmacro.org. Other programs for mediation, moderation, or conditional process analyses might be used instead</td>
		</tr>
		<tr>
			<td>Cognitive Competence Scale in the Survey of Attitudes towards Statistics (SATS-28)</td>
			<td>Candace Schau (Arizona State University)</td>
			<td></td>
			<td>In case it is used, request should be requested from the author, who whold the copyright</td>
		</tr>
		<tr>
			<td>Mastery Approach Scale in the Achievement Goal Questionnaire-Revised</td>
			<td>Andrew J. Elliot (University of Rochester)</td>
			<td></td>
			<td>In case it is used, request should be requested from the author</td>
		</tr>
		<tr>
			<td>Regulation of Cognition Scale of the Metacognitive Awareness Inventory</td>
			<td>Gregory Schraw (University of Nevada Las Vegas)</td>
			<td></td>
			<td>In case it is used, request should be requested from the creator</td>
		</tr>
	</tbody>
</table>

problem-solving before instruction (ps-i): a protocol for assessment and intervention in students with different abilities

Nowadays, how to encourage students' reflective thinking is one of the main concerns for teachers at various educational levels. Many students have difficulties when facing tasks that involve high levels of reflection, such as on STEM (Science, Technology, Engineering and Mathematics) courses. Many also have deep-rooted anxiety and demotivation towards such courses. In order to overcome these cognitive and affective challenges, researchers have suggested the use of "Problem-Solving before Instruction" (PS-I) approaches. PS-I consists of giving students the opportunity to generate individual solutions to problems that are later solved in class. These solutions are compared with the canonical solution in the following phase of instruction, together with the presentation of the lesson content. It has been suggested that with this approach students can increase their conceptual understanding, transfer their learning to different tasks and contexts, become more aware of the gaps in their knowledge, and generate a personal construct of previous knowledge that can help maintain their motivation. Despite the advantages, this approach has been criticized, as students might spend a lot of time on aimless trial and error during the initial phase of solution generation or they may even feel frustrated in this process, which might be detrimental to future learning. More importantly, there is little research about how pre-existing student characteristics can help them to benefit (or not) from this approach. The aim of the current study is to present the design and implementation of the PS-I approach applied to statistics learning in undergraduate students, as well as a methodological approach used to evaluate its efficacy considering students' pre-existing differences.

This protocol was satisfactorily implemented in a previous study23, with the exception of the measures of students&#39; predispositions in terms of their sense of competence, mastery approach goals, metacognition, and divergent thinking.
To address these predispositions, this protocol includes measures that have been previously validated and that have shown high levels of reliability (Table 1).
Typical solutions generated by ...

Watch this Scientific Journal Video about Problem-Solving Before Instruction PS-I: A Protocol for Assessment and Intervention in Students with Different Abilities at JoVE.com

Problem-Solving Before Instruction PS-I: A Protocol for Assessment and Intervention in Students with Different Abilities

This protocol guides researchers and educators through implementation of the Problem-Solving before Instruction approach (PS-I) in an undergraduate statistics class. It also describes an embedded experimental evaluation of this implementation, where the efficacy of PS-I is measured in terms of learning and motivation in students with different cognitive and affective predispositions.

Problem-Solving Before Instruction (PS-I): A Protocol for Assessment and Intervention in Students with Different Abilities

Nowadays, how to encourage students' reflective thinking is one of the main concerns for teachers at various educational levels. Many students have ...

A great proportion of the students find learning and motivational problems in courses that require critical thinking. This protocol is significant because it provides educators and researchers with guidelines to implement an approach that can be effective to face this challenge, the problem-solving before instruction approach. This approach consists on:before explaining a concept in class, giving the students the opportunity to invent personal solutions in relation to that concept.Also, this protocol is significant because it makes accessible and experimental evaluation about the efficacy of problem-solving before instruction, integrating this evaluation with the real educational practice, and attending to the variability of a student in terms of capacities and motivational predispositions. The protocol is contextualized in a statistics class of variability. Concretely, the problem-solving before instruction condition consists in having students to invent variability measures before they receive instruction about this topic.One of the reasons why we think it's important to make accessible the problem-solving before instruction approach into the educational practice is that it can help promote critical thinking. It is an opportunity for the students to face novel problems and try to find creative solutions for these problems. Another reason is that this experience is compatible with the teaching of the content in class.Specifically, several studies suggest that inventing personal solutions in these series of problem can help the students to activate previous knowledge, to become more aware of the knowledge gaps, to feel more motivated, and finally, to account a higher understanding of the contents covered and to be able to transfer this understanding to different situations. Nevertheless, it is a great importance that we continue investigating about the efficacy of problem-solving before instruction, especially because its implementation can generate negative reactions in some students. For example, some students can feel over challenged or even frustrated with the different options to consider in this first invention task.It is especially important that we evaluate how our students with different cognitive and motivational predispositions can benefit more or less from this approach. The protocol can be just used as a guide for implementing the problem-solving before instruction approach. However, if the user is also interested in the experiment evaluation before implementing the protocol, it should be explained to students the ethical considerations.It is important to warranty their free consent to participate by explaining them that if they do not want to participate in the investigation, they can perform the learning activities with a handing them in. Inform about ethical considerations verbally and with a written informed consent, allowing students to keep a copy of the informed consent. If students are below legal age, a parental informed consent should be requested.To warranty the anonymity of the data, randomly assign students an arbitrary identification number. Ask students to complete the different assessment tool to measure the cognitive and motivational predispositions. This assessment will prevent on the interest of the researcher.Nevertheless, in the protocol, it's described a proposal to measure the predispositions of previous academic knowledge, motivational goals, sense of competence, divergent thinking abilities, and metacognitive regulation abilities. After this assessment, students will be assigned to two learning conditions, the problem solving before instruction condition in which they problem solve in relation to the target concept before receiving instruction. Under direct instruction condition in which they problem-solve only after receiving some instructions.Have the task books, containing the materials, for the two conditions properly prepared. In this protocol, the task book for the problem-solving before instruction condition contains two activities. First, the problem where students invent variability measures.Second, a worked sample providing instructions about variability measures in the context of this problem. The task book for the direct instruction condition also contains two activities. First, instruction through the work example.Second, a practice problem where students applied the contents learned. To prevent the students to see the contents of the second activity while they are performing the first activity, attach together the papers corresponding to the second activity. Randomly assign the two task books to the students of the class.The assignments should not depend on how students are seated. After having the students work for 15 minutes in their first assigned activity, give them another 15 minutes to work in their second assigned activity. All instructions are written in the task books.It is recommended that a teacher is available to guide the students. However, it is important that during the invention phase, the teacher does not give the students any clues about the conventional solution because it can shortcut the development of their personal ideas. To help students explore the problem from their into deep ideas, the problem is designed based on the technique of contrasting cases.It means the data of the program are presented with some refined examples that students can easily compare side by side in a way that each comparison differs only in one or few relevant characteristics. If students need help, the teacher can guide them through metacognitive prompt, such as asking them to explain what type of solution they are trying to do. And by helping them identify the goal of the program by providing them with examples of how to generating a general procedure.To compliment the learning of both learning conditions, provide instructions through a lecture. For the specific context of this protocol, there is a document including animations and instructions proposed for this lecture. At the end of the lesson, ask students to complete the assessment for the dependent variables of interest.In the protocol, there is a proposal to measure curiosity filled by students and three types of knowledge performance, procedural knowledge, conceptually knowledge, and task for knowledge. Code for the final score in each assessment. For the assessments proposed in this protocol, guidelines are provided in the coding of the data section.The identification number will be used to connect all assessment belonging to each student. Perform the analysis of interest. Different guidelines are provided in the analysis of the data section in this protocol for the programs, SPSS and PROCESS.This images show common responses to the invention problem solved by students in the problem-solving before instruction condition. None of these solutions correspond to the target concept of the standard deviation. However, they are partial solutions and reveal reflections about important conceptual aspects.In contrast, the solutions in the practice problems of the direct instruction condition are more homogeneous and aligned with the canonical concept of the standard deviation because students solved this problem after receiving instruction with the worked example. An interesting result from the experimental evaluation of the protocol is the comparison between the problem solving before instruction condition and the direct instruction condition for each dependent variable considered. It can provide information about the general efficacy of problem-solving before instruction.Also, it can be interesting to explore the process that can mediate the effect of problem-solving before instruction or learning or the moderating role of students with disposition in this effects. For example, a potential moderation result can be that the efficacy of problem-solving before instruction depends on the students'metacognitive abilities. According to this graph, the students who have hight metacognitive skills benefit more from problem-solving before instruction, while those students with low metacognitive skills benefit more from direct instruction.In conclusion, this protocol can be useful to make the problem-solving before instruction approach and experimental evaluation more accessible to educators and researchers. This protocol has three important advantage. First of all, its flexibility to be adapted to the different educational and research interests.Secondly, its easy integration with the normal educational practice, and finally, the experimental evaluation considers the variability of students with different cognitive and motivational profiles. Problem-solving before instruction is a technique that can help to promote critical thinking, motivation, and deep learning. Investigating about its efficacy in different types of students can help educators to take useful decisions about its general implementation in educational practice.

Problem-Solving Before Instruction (PS-I): A Protocol for Assessment and Intervention in Students with Different Abilities

Abstract