The following protocol describes the development of an assessment rubric for the purposes of measuring student learning outcomes. The process is applied to a doctoral nursing education program.
<ol>
<li>Introduction. Over the past two decades, teachers in healthcare education have been increasingly challenged to demonstrate through their educational programs how students are adequately prepared to function in complex service environments upon graduation (1, 2, 3,4, 5, 6). The concept of a student learning outcome measures is built on a desire to demonstrate through a measurement process how a student who successfully completes the learning process related to an area of knowledge. It is different from one who does not complete the same learning process. Grades alone do not demonstrate a student s mastery of the learning outcome but rather successful or unsuccessful achievement in response to a testing procedure at a point in time. Focus on grading at the expense of learning contributes to culture of testing rather than a culture of learning (7). Further, most testing procedures measure various concepts quantitatively that have been or are easily quantifiable. But what about those aspects of a science or skill that relate more to quality? How should educators measure these aspects? How should they be reflected in expected student learning outcomes? How should they be quantified so they can be appropriately measured? As a society we do not typically value what we cannot measure. This protocol provides a method for quantifying concepts and constructs in any discipline or skill through the use of what are known as "rubrics"(8). Once these concepts are quantified, they can then be measured, altered, and refined to demonstrate that knowledge has been attained by the student.</li>
<li>Structure. The following core concepts are visually defined as they relate to the assessment process. First, the development of the concept of a learning outcome is illustrated as follows (see Table 1). A learning outcome is the embodiment of the knowledge a student should have upon successful completion of the learning process related to some type of content. Learning outcomes should encompass knowledge that is grounded in measurement (intrinsically quantitative) and knowledge that is grounded in meaning (intrinsically qualitative). 
<ol>
<li>Structure-methodology. The next core concept of the assessment process is measurement methodology. Many educators confuse successful attainment of student learning outcomes with successful scores on a grade. Use of a grade as the only measurement methodology only reflects half of the potential assessment of the learning process. Perhaps this is best illustrated with the following question, which student has successfully grasped the learning outcome, the one who scores a 90% on an exam but cannot actualize the knowledge in a real-world situation versus the student who scores a 70% on the same exam and can actualize the knowledge when needed in a real-world situation? Grades alone provide a baseline, point in time measurement of knowledge verification and synthesis. They do not provide measurement of knowledge translation or application.</li>

<li>Structure-rubric Definition. Now that we understand the underpinnings of a learning outcome, we need to understand the best way to measure it. One good way is through the use of a matrix or assessment rubric. To properly develop a student learning outcome assessment rubric, one must first define what a rubric is. It is a scoring tool that, when properly constructed, allows the person using it to assess a piece of work using a defined set of criteria. Further, it provides a mechanism for defining quality by moving those "gut perceptions" into measurable phenomena. This has been shown to be of valuable to both teacher and learner especially in adult learning environments (9, 10).</li>

<li>Structure-rubric Composition. A rubric typically takes the physical shape of a grid with rows and columns. The rows delineate the various categories that define the concept or construct that you are trying to measure while the columns represent the varying degrees of each criteria you are measuring within each category. Table 2 provides the framework for a basic rubric.</li>

<li>Structure-rubric Overview. Once you understand what a rubric is and its function, you will need to think about what those aspects are of the concept or construct that you are trying to define. Table 3 provides an example of quantitative and qualitative concepts that could be applied to and measured in a learning outcomes assessment rubric.</li>
</ol>
</li>
<li>Process. The method for developing a successful rubric is grounded in the rubrics' ability to quickly and accurately measure the concepts or constructs you want to assess. In its most rudimentary state, it is the embodiment of the assessor s "gut feeling". Ideally, the measurement process should take into account quantitative aspects (things that are routinely, easily measured) and qualitative aspects (things that have meaning, not easily measured) and quantify both in the form of an assessment rubric.</li>

<li>Outcome. The desired outcome of developing and using an assessment rubric in the education process is to accurately measure student learning outcomes. The learning outcomes are measurable statements of what the teacher expects the student to be able to know, do, value, and think about, etc. upon completion of the course or program.</li>

<li>Application. Harden noted that the learning outcomes could be quantified in a way that allows for measurement of competencies necessary for practice in medicine (11). Well designed learning outcomes that measure performance in capstone courses have been shown to be an effective tool in correlating nursing education preparation for successful transition into practice (12). Properly constructed rubrics to assess learning outcomes are a core component of a well designed learning outcomes measurement program (13, 14). They have significant application in professional education programs where qualitative concepts are central to practice. This protocol is applied to a doctoral nursing education program for the purposes of demonstrating how to develop and use the assessment rubrics in course and program level evaluation of desired student learning outcomes.
<ol>
<li>Application-scenario. The Doctor of Nursing Practice (DNP) is a relatively new doctoral degree. It is the practice doctorate for nursing. It is the sister doctorate to other practice doctorates in healthcare such as the MD for physicians, the PharmD for pharmacists, and the PsyD for psychologists. At its core, is the expectation of graduates to be able to apply techniques such as synthesis and ethical decision making for the purposes of knowledge translation of research into nursing practice. The accreditation organization that certifies DNP programs requires that student learning outcomes measure the ability of the student to demonstrate these competencies previously described and others through application of skills and knowledge at a course and program level.</li>

<li>Application-learning Outcome Development. A teacher in the DNP program wishes to teach the knowledge and skills to students so they can demonstrate the ability to make ethical decisions. First, the teacher should use the following question to guide their creation of a learning outcome measure: What is it that I would want to say distinguishes a student who has grasped the concept of X than someone who has not grasped the concept of X. In this application, the concept is ethical decision making (see Table 1 for example).</li>

<li>Application-concept Development. Next, the teacher should write down these concepts that underpin the structure of whatever the outcome is that you are trying to measure. In our case, ethical decision making is what we want to assess. Again, Table 1 provides an example of these key concepts.</li>

<li>Application-quantitative Rubric Section Development. Third, the teacher should determine which of these concepts are intrinsically quantitative and may be measured as learning outcomes that encompass knowledge grounded through measurement. Table 4 represents the quantitative measurements of ethical decision making in a rubric that would measure our proposed learning outcome.</li>

<li>Application-qualitative Rubric Section Development. Fourth, the teacher should determine which of these concepts are intrinsically qualitative and may be measured as learning outcomes that encompass knowledge grounded in meaning. This section of rubric development may be implemented in one of two ways: either as a pure-qualitative descriptor section rubric or a quasi-qualitative descriptor section of the rubric. Once this section of the rubric is completed, the rubric is ready for use in assessing learning outcomes.</li>

<li>Application-qualitative Rubric Section Development Explicated. It is important to describe in additional detail the importance of clarifying how a teacher intends to measure the qualitative concepts of a learning outcome. The benefit of the pure-qualitative level rubric is the articulation of previously ill-defined or un-defined concepts into words which helps a teacher begin to qualify "gut" perceptions of student learning. However, the price of using the pure-qualitative approach to rubric measurement is the sacrifice of a more reliable assessment rubric because the application of the score is still in the eye of the assessor. Conversely, development of the concepts into a quasi-qualitative descriptor assessment facilitates reliable measurement when it is used by other assessors because it quantifies the concepts in a simple, objectively measurable format.</li>

<li>Application-rubric Testing. Now that the rubric is created, it is important to determine how it will be applied to measure the outcome. In general, learning outcome measurement rubrics are applied to learning activities in a course and have a grade component associated with them. The example in Table 5 shows how the rubric we just created is applied to an ethical decision-making exercise.</li>
</ol></li>
</ol>

<ol>
	<li>Facione, N. C., Facione, P. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Assessment+design+issues+for+evaluation+of+critical+thinking+in+nursing.">Assessment design issues for evaluation of critical thinking in nursing.</a> Holistic Nursing Practice. 10 (3), 41-52 (1996).</li><li>Epstein, R. M., Hundert, E. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Defining+and+assessing+professional+competence.">Defining and assessing professional competence.</a> Journal of the American Medical Association. 287 (2), 226-235 (2002).</li><li>Stone, S. L., Qualters, D. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Course-based+assessment:+Implementing+outcome+assessment+in+medical+education.">Course-based assessment: Implementing outcome assessment in medical education.</a> Academic Medicine. 73 (4), 397-401 (1998).</li><li>Boland, D. L., Laidig, J., Palomba, C. A., Banta, T. W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Assessing+student+competence+in+accredited+disciplines:+Pioneering+approaches+to+assessment+in+higher+education.">Assessing student competence in accredited disciplines: Pioneering approaches to assessment in higher education.</a> Assessment of student learning in the discipline of nursing. , 71-94 (2001).</li><li>Epstein, R. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Assessment+in+medical+education.">Assessment in medical education.</a> New England Journal of Medicine. 356 (4), 387-396 (2007).</li><li>Halonen, J. S., Bosack, T., Clay, S., McCarthy, M., Dunn, D. S., Hill, G. W., McEntarffer, R., Mehrotra, C., Nesmith, R., Weaver, K. A., Whitlock, K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+rubric+for+learning,+teaching,+and+assessing+scientific+inquiry+in+psychology.">A rubric for learning, teaching, and assessing scientific inquiry in psychology.</a> Teaching of Psychology. 30 (3), 196-208 (2003).</li><li>Birenbaum, M., Douchy, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Alternative in assessment of achievements, learning processes and prior knowledge. , (1996).</li><li>Knowles, M. S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Modern Practice of Adult Education: From Pedagogy to Andragogy. , (1980).</li><li>Montgomery, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Authentic+Tasks+and+Rubrics:+Going+Beyond+Traditional+Assessments+in+College+Teaching.">Authentic Tasks and Rubrics: Going Beyond Traditional Assessments in College Teaching.</a> College Teaching. 50 (1), 34-39 (2002).</li><li>Wolf, K., Stevens, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+role+of+rubrics+in+advancing+and+assessing+student+learning.">The role of rubrics in advancing and assessing student learning.</a> Journal of Effective Teaching. 7 (1), 3-14 (2007).</li><li>Rebeschi, L. A. r. o. n. s. o. n., B, . <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Assessment+of+Nursing+Student's+Learning+Outcomes+and+Employment+Choice+after+the+Implementation+of+a+Senior+Capstone+Course.">Assessment of Nursing Student's Learning Outcomes and Employment Choice after the Implementation of a Senior Capstone Course.</a> International Journal of Nursing Education Scholarship. 6 (1), 1-15 (2009).</li><li>Bolton, F. C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Rubrics+and+Adult+Learners:+Andragogy+and+Assessment.">Rubrics and Adult Learners: Andragogy and Assessment.</a> Assessment Update. 18 (3), 5-6 (2006).</li><li>Andrade, H. G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Teaching+with+rubrics:+The+good,+the+bad,+and+the+ugly.">Teaching with rubrics: The good, the bad, and the ugly.</a> College Teaching. 53 (1), 27-30 (2005).</li></ol>

No conflicts of interest declared.

<ol>
<li>We have demonstrated what a learning outcome rubric is, how to construct one, and how to use it in the assessment of learning outcomes by applying it to a DNP education program. There are several key points that should be further described at this point.
<ol>
<li>Learning Outcome Rubrics versus Grading Rubrics. First, it is important to delineate the difference between a grading rubric and a learning outcomes rubric. A grading rubric is designed to help faculty assess what grade a student should receive based on their performance on a particular assignment and may or may not involve both quantitative and qualitative aspects, whereas a learning outcomes rubric is focused on helping faculty assess if a student learned all of the key and critical aspects of a concept or construct. The measurement of the learning outcome typically includes quantitative and qualitative aspects. Most importantly, an assignment that is graded is usually part of the overall assessment of a learning outcome.</li>
<li>Quantification of Intrinsically Qualitative Concepts. Once you understand the difference between a learning outcome rubric and a grading rubric, the most challenging part of constructing the learning outcome assessment rubric is determining to what level you wish to define the qualitative components. As mentioned previously, leaving the definition of components at a pure-qualitative level allows individual teacher freedom in assessing success or failure however; it is not usually reliable for use by other teaching faculty and as its definitions are often in the eye of the assessor. Further, this type of rubric also allows for student challenges of grading and outcomes assessment since again its interpretation may be in the eye of the beholder.</li>

<li>Course Level versus Program Level Learning Outcomes. Use of properly constructed learning outcome assessment rubrics allow for proper course level evaluation of teaching and learning processes and procedures. However, these rubrics are also applicable to program level evaluation of learning outcomes. For example, the results from a student cohort s scores on a particular learning outcome such as the one used in the application could be used to demonstrate the effectiveness of the teaching and learning process at an aggregate or program level. Then, these results can be used to help a program s faculty clearly track exactly which areas of the teaching process need to be modified as part of a learning program s continuous performance improvement process.</li>
</ol></li>
<li>Conclusion. We have attempted to explicate the concepts related to learning outcome rubrics; specifically, what they are and are not, how to construct them and how to use them. Further, we have applied the development of learning outcomes using a case study example of a DNP program. Learning outcome measurement represents an often misunderstood concept and misapplied process that can result in teaching faculty and student learner dissatisfaction. This presentation focused on demonstration of the development of a temporal knowledge through visual articulation of learning outcome measurement constructs.</li>
</ol>

using learning outcome measures to assess doctoral nursing education

Education programs at all levels must be able to demonstrate successful program outcomes. Grades alone do not represent a comprehensive measurement methodology for assessing student learning outcomes at either the course or program level. The development and application of assessment rubrics provides an unequivocal measurement methodology to ensure a quality learning experience by providing a foundation for improvement based on qualitative and quantitatively measurable, aggregate course and program outcomes.  Learning outcomes are the embodiment of the total learning experience and should incorporate assessment of both qualitative and quantitative program outcomes. The assessment of qualitative measures represents a challenge for educators in any level of a learning program. Nursing provides a unique challenge and opportunity as it is the application of science through the art of caring. Quantification of desired student learning outcomes may be enhanced through the development of assessment rubrics designed to measure quantitative and qualitative aspects of the nursing education and learning process. They provide a mechanism for uniform assessment by nursing faculty of concepts and constructs that are otherwise difficult to describe and measure. A protocol is presented and applied to a doctoral nursing education program with recommendations for application and transformation of the assessment rubric to other education programs. Through application of these specially designed rubrics, all aspects of an education program can be adequately assessed to provide information for program assessment that facilitates the closure of the gap between desired and actual student learning outcomes for any desired educational competency.

Watch this Scientific Journal Video about Using Learning Outcome Measures to assess Doctoral Nursing Education at JoVE.com

Using Learning Outcome Measures to assess Doctoral Nursing Education

The structure and process for measuring student learning outcomes through customizable assessment rubrics is discussed and applied to a doctoral nursing education program.

Education programs at all levels must be able to demonstrate successful program outcomes. Grades alone do not represent a comprehensive measurement ...

using-learning-outcome-measures-to-assess-doctoral-nursing-education

Research

Education

JoVE Core

JoVE Journal

Nursing

Medicine

Unit 1

The Nursing Process III

SCIENTISTS IN ACTION

18.8K Views.  Harris College of Nursing and Health Sciences. The structure and process for measuring student learning outcomes through customizable assessment rubrics is discussed and applied to a doctoral nursing education program.