Discussion and Plan

Overview

Our discussion begins with a quick recount of our process for conducting the Inquiry Brief. First, in September 2007 during a School of Education, Leadership and Public Service faculty meeting, the NCATE Leadership Team approached the faculty with the idea of switching our accreditation from NCATE to TEAC. Upon School of Education, Leadership and Public Service faculty approval, the leadership team initiated the formal process of requesting approval internally at NMU. Finally, on December 14, 2007 the NMU Board of Regents approved our decision making the "internal process" transition official.

Over the next five months, each faculty meeting contained "TEAC Updates", whereby faculty approved each step of the process. In other words, faculty approved the switch to TEAC, then approved the Claims, then approved the Data sources, then participated in reviewing the results, creating "Discussions" documents linked below, and finally on October 16th, our entire Teacher Education Unit was represented and endorsed the document. In what follows, we outline in more detail the process by which faculty reviewed the claims and created "Discussion" documents.

Under the direction of Joe Lubig, the Secondary Education Committee made up of faculty members representing the Teacher Education Unit met on Friday, September 12th, 2008. Under the direction of K.C. Holder, the School of Education, Leadership and Public Service Faculty met on Thursday, September 18th, 2008. Faculty members were charged to analyze and evaluate the results of the data sources that were pertinent to the respective committees. As an aid to "standardize" the process across reviewers, we created an "Analysis Table" (see below) for each claim for faculty to use to guide their process.

 


Data Source

What?

So What?

Now What?

Student Teaching Evaluation Form

     

...

     

In teams of two or three, faculty analyzed each individual data source under each claim and our program-wide survey quantitative data by completing the Analysis Table. After completing the table, the faculty team was prompted with "Upon review of the claim we found...", then asked to review their own "Now What?" column and write a supporting narrative based upon their analysis. Further, on September 25th, 2008 faculty teams of two to three analyzed the four survey qualitative open-ended results holistically by looking for "themes" across the results and provided descriptive accounts of that data source. Detailed faculty discussion and plans for program improvement beyond the summary presented here can be found in Appendix G:

Claim 1 - Appendix G

Claim 2 - Appendix G

Claim 3 - Appendix G

Claim 4 - Appendix G

Claim 5 - Appendix G

Program-wide Surveys and Evaluations

Open-ended survey items

     What is an effective teacher?

     Please comment on 3 areas of more / less emphasis

     Please provide any additional comments regarding teacher preparation at NMU

     Please comment on candidates' strengths and weaknesses (submitted by cooperating     teachers)

Reliability and Validity Revisited

Upon the near completion of our Inquiry Brief, we return to the issues of reliability and validity to "member check" our process and now the product of our NMU Inquiry Brief. In general, we tended to look at this Inquiry Brief as a utilization-focused evaluation of our program rather than inquiry as conducting research to publish for a journal. More specifically, we aligned with the Accuracy Standards of the Joint Committee on Standards for Educational Evaluation (1994) that describe reliable and valid information as:

  • Reliability Information: The information-gathering procedures should be chosen or developed and then implemented so that they will assure that the information obtained is sufficiently reliable for the intended use (P. A6).
  • Valid Information: The information-gathering procedures should be chosen or developed and then implemented so that they will assure that the interpretation arrived at is valid for the intended use (P. A5).

Further, we also agree with Patton (2008) that, "The emphasis is on appropriateness and credibility – measures, samples, and comparisons that are appropriate and credible to address key evaluation issues" (p. 394). With this, we now turn to a re-examination of the reliability of measures followed by the validity of our Inquiry Brief.

Regarding "reliability"

Near the completion of our Inquiry Brief, regarding the reliability of measurements, we reviewed our Claims and Rationale section where we described how the Student Teaching Evaluation Form, Grades, Michigan Test for Teacher Competence, and anecdotal data sources were viewed. At this stage, the only data source (i.e., measure) we felt needed a more detailed look regarding reliability was our Student Teaching Evaluation Form.

In what follows, we present three distinctly different "views of reliability" of the Elementary Claim 1 data and part of the Elementary Claim 4 data.  More specifically, we describe the data using a Correlation Matrix, Graphs, and finally a Table describing the Percentage Agreement.

We selected Claim 1 as a starting point, and "Claim 4_Instructional Competence" as a second area to explore. We felt that we needed to better understand the consistency of ratings across the raters. We asked the question, "How do our raters stack up against each other?" Although we recognized our data was percentages rather than paired ratings, using Exell, we created a correlation matrix of the Claim 1 and Claim 4 data to analyze. Tables 1.26 and 1.37 display the correlation matrix data.

 

Table 1.26: Claim 1_ELEMENTARY_Correlation Matrix


Claim 1_ELEMENTARY_Correlation Matrix

 

Student Self

Cooperating Teacher

University Supervisor

 

P

I

B

P

I

B

P

I

B

 

Column 1

Column 2

Column 3

Column 4

Column 5

Column 6

Column 7

Column 8

Column 9

Column 1

1.00

               

Column 2

 

1.00

             

Column 3

   

1.00

           

Column 4

0.68

   

1.00

         

Column 5

 

0.85

   

1.00

       

Column 6

   

#DIV/0!

   

1.00

     

Column 7

0.71

   

0.31

   

1.00

   

Column 8

 

0.70

   

0.47

   

1.00

 

Column 9

   

-0.22

   

#DIV/0!

   

1.00

Table 1.27: Claim 4_ELEMENTARY_Instructional Competence_Correlation Matrix
(Note: Due to spacing of the page, columns 9-12 are removed from the horizontal. These columns represent the "University Supervisor" ratings of P, I, B, and NC.)


Claim 4_Elementary_Instructional Competence_Inter-rater Reliability Matrix

 

Student Self Report

 

Cooperating Teacher

 
 

P

I

B

NC

P

I

B

NC

 

Column 1

Column 2

Column 3

Column 4

Column 5

Column 6

Column 7

Column 8

Column 1

1.00

             

Column 2

 

1.00

           

Column 3

   

1.00

         

Column 4

     

1.00

       

Column 5

0.71

     

1.00

     

Column 6

 

0.73

     

1.00

   

Column 7

   

#DIV/0!

     

1.00

 

Column 8

     

0.11

     

1.00

Column 9

0.63

     

0.50

     

Column 10

 

0.73

     

0.67

   

Column 11

   

#DIV/0!

     

#DIV/0!

 

Column 12

     

0.11

     

0.11

NOTE:  Analysis and evaluation of Table 1.29 and Table 1.30 should be based upon recognizing that the data used was percentages of raters by respondents rather than paired samples.

Upon review of the correlation matrix, we decided the second approach to describing the reliability of our data was to visually display the same data sets.  Figure 1.2, 1.3, and 1.4 displays the line graph of the corresponding data.

Figure 1.2: Claim 1_ELEMENTARY_% Proficient

Claim 1 Elem Prof

Figure 1.3: Claim 4_ELEMENTARY_Instructional Competence_% Proficient

Claim 4

Figure 1.4: Claim 4_ELEMENTARY_Instructional Competence_% of "No Comment"

Claim 4

The graphs of our data, when looked at using the same vertical scale (i.e., 0-100%) clearly indicate that there is exceptionally high agreement from each rater across all of the indicators.  Further, knowing that our data was not originally paired data or even in a form for calculating reliability statistics we felt the correlation matrix did provide a lower than actual description of the relationship across raters..

Upon review of the correlation matrixes (especially the very low numbers between cooperating teacher and university supervisor) and the graphs, we decided to further explore the relationship between raters by calculating the percentage of observer agreement within Claim 1 using the following formula:


formula

This formula was chosen because our original data set was recorded in percentage of respondents which we understood to skew the correlation matrix. In other words, we felt the correlation matrix was not as high as the actual rater agreement simply because the data was percentage values rather than paired ratings.

Albeit in what is probably an over inflation, the percentage of observe agreement (Table 1.28) displays a much better "relationship" between the students, cooperating teachers and university supervisor ratings. As we continued to analyze the student teaching evaluation data, and in particular the data we had access to, we felt using the Graph figures (Figures 1.2, 1.3, and 1.4) was the best representation and the percentage of observer agreement was also used to help indicate areas for future study.

 

Table 1.28: Percentage of Observer Agreement for Claim 1

 

Student to
Cooperating
Teacher

Student to
University
Supervisor

Cooperating
Teacher to
University
Supervisor

A. KNOWLEDGE

     

A.1 Content Areas

97.7%

98.0%

99.7%

A.2 General Knowledge

99.3%

98.8%

99.4%

       

B. INSTRUCTIONAL COMPETENCE

     

B.1 Communication Skills

96.0%

96.8%

99.2%

B.2 Management of Students' Behavior

93.1%

91.4%

98.3%

B.3 Management of Learning Progress

96.4%

95.7%

99.2%

B.4 Management of  
      Contingencies/Emergencies

99.3%

98.7%

99.4%

B.5 Lesson/Unit Planning

96.6%

97.0%

99.6%

B.6 Lesson/Unit Presentation

97.1%

97.5%

99.6%

B.7 Use of Materials and Resources

97.1%

97.0%

99.9%

       

C. ASSESSMENT COMPETENCE

     

C.1 Selection, Creation,
      and Use of Student Assessments

93.8%

94.6%

99.2%

C.2 Grading and
      Reporting Student Achievement

96.5%

98.3%

98.2%

C.3 Evaluation of Teaching, Materials,
      and Curriculum

97.1%

96.0%

99.0%

       

D. PROFESSIONALISM

     

D.1 Professional Ethics

99.8%

99.3%

99.4%

D.2 Professional Attitude

99.3%

99.3%

99.9%

D.3 Professional Development/Service

96.0%

97.2%

98.8%

D.4 Knowledge and Execution of Duties

96.1%

96.5%

99.6%

D.5 Knowledge of the School
      and its Context

96.5%

95.9%

99.4%

D.6 Human Relations

99.8%

99.3%

99.4%

Regarding "validity”

Given our "faculty consensus" approach, the validity of our overarching claims and our specific detailed measures and interpretations resides in our approach where "multiple eyes" were consistently examining the process and products at each stage of the Inquiry Brief. The appropriateness and credibility of each individual measure (or section of the brief) was being "member checked" at each of the progressing stages of the brief. This constant presentation / clarification / consensus cycle, we believe, ensured that the validity of our overarching claims interpretations grew out of valid inferences from individual measures. In other words, valid interpretations grew at each stage of the process. Thus, validity was not only credible, but enhanced due to the design of our approach to conducting the inquiry brief.

More specifically, and using traditional validity checks (Creswell (2008); Fraenkel & Wallen (2006)), the table that follows briefly describes validity checks for our completed Inquiry Brief.


Validity Check

Explanation

Content Validity

Content validity was enhanced by alignment. First, Claim 1
accounted for a holistic sampling of how our students do overall on the state standards and on our specific student teaching evaluation; whereas, Claims 2-5 account for specific and detailed indicators that were aligned to the content of the specific claim from the student teaching form. For example, the specific indicators from the evaluation form that pertain to integrating technology were collected, analyzed, discussed in regards to our Claim 2 (student integrate technology). In other words, content validity was purposeful by the design and alignment of our data sources.

Criterion-related validity

Our approach to criterion-related validity was to examine the convergence of concurrent measures.  We clearly recognize that our measures (i.e., data sources) within our Inquiry Brief were not meant to serve as predictive variables.  Thus, we used criterion-related validity in the sense of examining the strength of the relationship among multiple measures. With that, each of our five claims contains a minimum of one standardized measure that is aligned to the external state standards (e.g., student teaching evaluation form); and, all of the candidates complete the measure.  Further, anecdotal measures were designed as claim specific and aligned to the claims. Due to our analysis review process (i.e., faculty teams working on distinct claims data) "convergence of multiple measures" was purposefully designed by having faculty describe "What?", "So what?", and "Now what?" for each data source under each claim.

Construct validity

We used construct-validity in the sense that the evidence must meet: (1) variables being measured are clearly defined; (2) claims based upon current requirements, published literature regarding the program; and, (3) the claims were justified. Perhaps our strongest evidence of validity is to examine our Inquiry Brief in regards to construct validity. We found that our Claims and Data Sources were clearly defined. Further, when viewing our claims holistically we found that all of our measures point in the same direction: Our data clearly document that our claims are justified.

Example Program changes

Over the past fifteen plus years, NMU Teacher Education consistently revised programs, documents, and practices based upon data. For example:

  • From 1989 to 2006, the current Director of the School of Education, Leadership and Public Service was the Director of Field Experiences. In his past role, and currently we surveyed our Candidates at the end of student teaching each semester and asked them to evaluate our program. Separate forms were used for elementary, secondary and special education students. Students were asked to rate how satisfied they were with various aspects of their program, including their required courses, and whether they rated themselves as at the advanced, proficient basic or awareness level in the seven areas on the entry- level standards. The Director of Field Experiences reported these findings to the faculty at large and used it to make improvements in the program. These data are available for review, but because some of the ratings and comments may be misused, we are not making them public. Overall our students rated themselves very high on the entry level standards and as being satisfied or very satisfied with their courses and program. In cases where courses were rated lower than others, we consulted as to how we might address these perceive areas of weakness.
  • The student teaching evaluation form itself has evolved through multiple stages along with the state revisions to the Entry-Level Standards for Michigan Teachers (see Rationale for further description of the development of this instrument).
  • In 2006-2007, the NMU's secondary history education program submitted a program change based upon students scored slightly below the state average on the Michigan Tests for Teacher Certification (MTTC). For more details see Secondary History education proposal.

Next steps

One "next step" is a given. On April 28, 2008, the State Superintendent of Public Instruction, Dr. Michael P. Flanagan, recommended to the State Board of Education to approve the Professional Standards for Michigan Teachers (PSMT). The PSMT simply constitute the fourth iteration of standards. The original ELSMT were developed in 1993, revised in 1998 and again in 2002. At present, NMU (as well as all institutions) will be required to update materials given the new PSMT. Along side of the standards a companion document, The Criteria for Assessment of Pedagogy for Initial Certification, was also created and revised. The newest iteration of the assessment document is called the Profile of Teacher Knowledge and Skills (PTKS). Institutions will have until 2011 to revise materials to meet the new PSMT and PTKS documents.

As we continue to move toward increased collaboration with all parties in the teacher education process, we expect more involvement by all in the development and improvement of our programs. This trend has grown over the last two decades and continues to develop. We were first in Michigan to incorporate the ELSMT into our program standards, even before they were officially adopted by the state and our incorporating the minor changes introduced in the PSMT into our assessment instruments will be relatively easy.

 

Conclusion

Upon faculty construction of claims, review of claims, review of data sources, analysis of the findings, the following concluding belief statements were generated:

  1. We believe our claims are valid representations of outcomes for our program.
  2. We believe all of our candidates for teacher certification met our claims.
  3. We believe our program needs to "study" relative emphasis on classroom management; and in particular, effectively teaching students with special needs. Candidate self-evaluations on indicators of competence in classroom management were consistently lower than cooperating teacher and/or university supervisor indicators. Please note: this did not indicate a program problem; rather, given the inconsistency we understand that further study is necessary.
  4. We believe our program needs to "study" our field experience evaluation instruments we use prior to our Final Student Teaching Evaluation Form (e.g., ED 230/231; Block 1; Block 2, etc.). We believe that our field experiences are powerful learning experiences for our candidates; however, the actual measurement used to document progress during field experiences is not where we want it to be.

Our growing consultation among the teacher education faculty, along with increased use of assessment data, allows us multiple opportunities to keep our program at the level of excellence we have aspired to and taken pride in. We are continually increasing our use of technology to facilitate learning and that trend promises to continue. Our design, development, implementation, and write-up of the NMU Inquiry Brief 2008 provided written justification and documentation of excellence across all programs within our Northern Michigan University Teacher Education Unit.

References

American Association of Colleges for Teacher Education. (2008). Handbook of technological
pedagogical content knowledge (TPCK) for educators. New York: Published by
Routledge for the American Association of Colleges for Teacher Education.

Banks, J. A., & Banks, C. A. M. (2004). Multicultural education: Issues and perspectives.
Wiley/Jossey-Bass education. Hoboken, N.J.: Wiley.

Creswell, J.W. (2008). Educational research: Planning, Conducting, and Evaluating
Quantitative and Qualitative Research (3rd ed.). Upper Saddle River, NJ: Merrill Prentice Hall.

Eisner, E. W. (1998). The enlightened eye: Qualitative inquiry and the enhancement of
educational practice. Upper Saddle River, N.J.: Merrill.

Fraenkel, J.R. & Wallen, N.E. (2006). How to Design and Evaluate Research in Education (6th
Ed.). Boston: McGraw-Hill.

Joint Committee on Standards for Educational Evaluation (1994). The Program Evaluation
Standards. Thousand Oaks, CA: Sage. http://www.wmich.edu/evalctr/jc.

Patton, M.Q. (2008). Utilization-focused Evaluation. (4th Ed.). Los Angeles, CA: Sage
Publications.

Scheffler, I. (1965). Conditions of knowledge; An introduction to epistemology and education.
Chicago: Scott, Foresman.

Scheffler, I. (1976). "Basic mathematics skills: Some philosophical and practical remarks."
Teachers College Record, 78(2), 205-212.

Schön, D. A. (1983). The reflective practitioner: How professionals think in action. New York:
Basic Books.

Shulman, L. S., & Wilson, S. M. (2004). The wisdom of practice: Essays on teaching, learning,
and learning to teach. San Francisco: Jossey-Bass.