This work was supported by a project of the Principality of Asturias (FC-GRUPIN-IDI/2018/000199) and a predoctoral grant from the Ministry of Education, Culture, and Sports of Spain (FPU16/05802). We would like to thank Stephanie Jun for her help editing the English in the learning materials.

This protocol follows the Helsinki Declaration of Ethical Principles for Research with Humans, but applies these principles to the added difficulties of integrating research within real-life settings in education32. Specifically, neither the assignment of learning conditions nor the decision to participate can have consequences for students&#39; learning opportunities. In addition, confidentiality and the anonymity of students is maintained even when it is the teachers who are in charge of the eva...

The aim of this protocol is to guide researchers and educators in the implementation and evaluation of the PS-I approach in real classroom contexts. According to some previous experiences, PS-I can help promote deep learning and motivation in students19,21,24, but there is a need for more research about its efficacy in students with different abilities and motivational predispositions14,<sup cl...

One of the questions that teachers are most concerned about currently is how to stimulate students&#39; reflection. This concern is common in courses of a mathematical nature, such as STEM courses (Science, Technology, Engineering and Mathematics), in which the abstraction of many concepts requires a high degree of reflection, yet many students report approaching these courses purely through memory-based methods1. In addition, students often show superficial learning of the concepts1,2,3. The difficulties that students experience applying reflection and deep learning processes, however, are not only cognitive. Many students feel anxiety and demotivation faced with these courses4,5. In fact, these difficulties tend to persist throughout students&#39; educations6. It is therefore important to explore educational strategies that motivationally and cognitively prepare students for deep learning, regardless of their differing predispositions.
It is particularly useful to find strategies that complement typical instructional approaches. One of the most typical being direct instruction. Direct instruction means fully guiding students from the introduction of novel concepts with explicit information about these concepts, then following that with consolidation strategies such as problem-solving activities, feedback, discussions, or further explanations7,8. Direct instruction can be effective for easily transmitting content8,9,10. However, students often do not reflect on important aspects, such as how the content relates to their personal knowledge, or potential procedures that could work and do not11. It is therefore important to introduce complementary strategies to make students think critically.
One such strategy is the Problem-Solving before Instruction (PS-I) approach12, also referred to as the Invention approach11 or the Productive Failure approach13. PS-I is different to direct instruction in the sense that students are not directly introduced to the concepts, instead there is a problem-solving phase prior to the typical direct instruction activities in which students seek individual solutions to problems before getting any explanation about procedures for solving them.
In this initial problem, students are not expected to fully discover the target concepts13. Students may also feel cognitive overload14,15,16 and even negative affect17 with the uncertainty and the many aspects to consider. However, this experience can be productive in the long term because it can facilitate critical thinking about important features. Specifically, the initial problem can help students to become more aware of the gaps in their knowledge18,activate prior knowledge related to the content to cover13, and increase motivation because of the opportunity to base their learning on personal knowledge7,17,19.
In terms of learning, the effects of PS-I are generally seen when the results are evaluated with deep learning indicators20,21. In general no differences have been found between students who learned through PS-I and those who learned through direct instruction in terms of procedural knowledge20,22, which refers to the ability to reproduce learned procedures. However, students who go through PS-I generally exhibit higher learning in conceptual knowledge7,19,23, which refers to understanding the content covered, and transfer7,15,19,24, which refers to capacity to apply this understanding to novel situations. For example, a recent study in a class about statistical variability showed that students who were given the opportunity to invent their own solutions to measure statistical variability before receiving explanations about the general concepts and procedures in this topic demostrated better understanding at the end of the class than those who were able to directly study the relevant concepts and procedures before getting involved in any problem-solving activity23. However, some studies have shown no differences in learning16,25,26 or motivation19,26 between PS-I and direct instruction alternatives, or even better learning in direct instruction alternatives14,26, and it is important to consider potential sources of variability.
The design features underlying the implementation of PS-I are an important feature20. A systematic review20 found that there was more likely to be a learning advantage for PS-I over direct instruction alternatives when the PS-I interventions were implemented with at least one of two strategies, either formulating the initial problem with contrasting cases, or building the subsequent instruction with detailed feedback about the students&#39; solutions. Contrasting cases consist of simplified examples that differ in a few important characteristics11 (see Figure 1 for an example), and can help students identify relevant features and evaluate their own solutions during the initial problem11,20. The second strategy, providing explanations that build on the students&#39; solutions13, consist of explaining the canonical concept while giving feedback about the affordances and limitations of solutions generated by students, which can also help students focus on relevant features and evaluate the gaps in their own knowledge20, but after the initial problem-solving phase is completed (see Figure 3 for an example of the scaffolding from students&#39; typical solutions).
Given the support in the literature for these two strategies, contrasting cases and building instruction on students&#39; solutions, it is important consider them when promoting the inclusion of PS-I in real educational practice. This is the first goal of our protocol. The protocol provides materials for a PS-I intervention that incorporate these two principles. It is a protocol that,&#160;while adaptable, it is contextualized for a lesson on statistical variability, a very common lesson for university and high school students, who are generally the target populations in the literature on PS-I29. The initial problem-solving phase consists of inventing variability measures for income distributions in countries, which is a controversial topic30 that may be familiar to students in many learning areas. Then materials are provided for students to study solutions to this problem in a worked example, and for a lecture that incorporates discussion of common solutions produced by students along with embedded practice problems.
The second goal of our protocol is to make the experimental evaluation of PS-I accessible to educators and researchers, which can facilitate the investigation of PS-I from a greater variety of perspectives while maintaining some conditions constant across the literature. Yet conditions of this experimental evaluation are flexible to modifications.&#160;The experimental evaluation described in the protocol can be applied in ordinary lessons, since students in a single class can be assigned the materials for the PS-I condition or the materials for a direct instruction condition at the same time (Figure 4). This direct instruction condition is also adaptable to research and education needs, but as originally described in the protocol students start by getting the initial explanations about the target concept with the worked example, and then consolidate this knowledge with a practice problem (only presented in this condition to compensate for the time PS-I students spend on the initial problem), and with the lecture23. Potential adaptations include starting with the lecture and then having students to do the problem-solving activity, which is a typical control condition for comparing PS-I that has often led to better learning for the PS-I condition7,13,19,26. Alternatively, the control condition can be reduced to the exploration of a worked example followed by the lecture phase, which, although a more simplified version of direct instruction approaches than originally proposed, is more common in the literature and has led to varied results, with some studies indicating better learning in PS-I15,24, and others indicating better learning from this type of direct instruction condition14,26.
Finally, a third goal of the protocol is to provide resources for evaluating how students with different predispositions and cognitive abilities can benefit from PS-I15. The evaluation of these predispositions is especially important if we consider the negative predispositions that some students often have with STEM courses, and the fact that PS-I can still produce negative reactions in some cases14. There is, however, little research on this.
On the one hand, since PS-I facilitates the association of learning with individual ideas, rather than just formal knowledge, PS-I can be hypothesized as being able to help motivate students from low academic levels, those who have low feelings of competence, or low motivation about the subject13,27. One study showed that students with low mastery orientation, i.e., fewer goals related to personal learning, benefited more from PS-I than those with higher motivation to learn27. On the other hand, students with other profiles might encounter difficulties when involved in PS-I. More specifically, metacognition plays an important role in PS-I31, and students with low metacognition skills might not benefit from PS-I due to difficulties in being aware of their knowledge gaps or discerning relevant content15. In addition, as the initial phase of PS-I is based on the production of individual solutions, students with low divergent abilities, difficulties generating a variety of responses in a given situation, might benefit less from PS-I than other students. The protocol presents reliable instruments to assess for these predispositions (Table 1) although others may be considered.
In summary, this protocol aims to make an implementation of a PS-I intervention that follows accepted principles in the PS-I literature accessible to educators and researchers. Additionally, the protocols provide an experimental evaluation of this intervention, and facilitate the evaluation of students&#39; cognitive and motivational predispositions. It is a protocol that does not require access to new technologies or specific resources, and one that can be modified based on research and educational needs.

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>SPSS Program</td>
			<td>International Business Machines Corporation (IBM)</td>
			<td></td>
			<td>Other programs for general data analysis might be used instead</td>
		</tr>
		<tr>
			<td>PROCESS program</td>
			<td>Andrew F. Hayes (Ohio State University)</td>
			<td></td>
			<td>Freely accesible at: http://www.processmacro.org. Other programs for mediation, moderation, or conditional process analyses might be used instead</td>
		</tr>
		<tr>
			<td>Cognitive Competence Scale in the Survey of Attitudes towards Statistics (SATS-28)</td>
			<td>Candace Schau (Arizona State University)</td>
			<td></td>
			<td>In case it is used, request should be requested from the author, who whold the copyright</td>
		</tr>
		<tr>
			<td>Mastery Approach Scale in the Achievement Goal Questionnaire-Revised</td>
			<td>Andrew J. Elliot (University of Rochester)</td>
			<td></td>
			<td>In case it is used, request should be requested from the author</td>
		</tr>
		<tr>
			<td>Regulation of Cognition Scale of the Metacognitive Awareness Inventory</td>
			<td>Gregory Schraw (University of Nevada Las Vegas)</td>
			<td></td>
			<td>In case it is used, request should be requested from the creator</td>
		</tr>
	</tbody>
</table>

problem-solving before instruction (ps-i): a protocol for assessment and intervention in students with different abilities

Nowadays, how to encourage students' reflective thinking is one of the main concerns for teachers at various educational levels. Many students have difficulties when facing tasks that involve high levels of reflection, such as on STEM (Science, Technology, Engineering and Mathematics) courses. Many also have deep-rooted anxiety and demotivation towards such courses. In order to overcome these cognitive and affective challenges, researchers have suggested the use of "Problem-Solving before Instruction" (PS-I) approaches. PS-I consists of giving students the opportunity to generate individual solutions to problems that are later solved in class. These solutions are compared with the canonical solution in the following phase of instruction, together with the presentation of the lesson content. It has been suggested that with this approach students can increase their conceptual understanding, transfer their learning to different tasks and contexts, become more aware of the gaps in their knowledge, and generate a personal construct of previous knowledge that can help maintain their motivation. Despite the advantages, this approach has been criticized, as students might spend a lot of time on aimless trial and error during the initial phase of solution generation or they may even feel frustrated in this process, which might be detrimental to future learning. More importantly, there is little research about how pre-existing student characteristics can help them to benefit (or not) from this approach. The aim of the current study is to present the design and implementation of the PS-I approach applied to statistics learning in undergraduate students, as well as a methodological approach used to evaluate its efficacy considering students' pre-existing differences.

This protocol was satisfactorily implemented in a previous study23, with the exception of the measures of students&#39; predispositions in terms of their sense of competence, mastery approach goals, metacognition, and divergent thinking.
To address these predispositions, this protocol includes measures that have been previously validated and that have shown high levels of reliability (Table 1).
Typical solutions generated by ...

Watch this Scientific Journal Video about Problem-Solving Before Instruction PS-I: A Protocol for Assessment and Intervention in Students with Different Abilities at JoVE.com

Problem-Solving Before Instruction PS-I: A Protocol for Assessment and Intervention in Students with Different Abilities

This protocol guides researchers and educators through implementation of the Problem-Solving before Instruction approach (PS-I) in an undergraduate statistics class. It also describes an embedded experimental evaluation of this implementation, where the efficacy of PS-I is measured in terms of learning and motivation in students with different cognitive and affective predispositions.

Problem-Solving Before Instruction (PS-I): A Protocol for Assessment and Intervention in Students with Different Abilities

Nowadays, how to encourage students' reflective thinking is one of the main concerns for teachers at various educational levels. Many students have ...