Research & Policy Briefs
TDOE researchers study department initiatives to determine the effectiveness of our work. In addition to sharing this research with the public, department leadership uses it to make future TDOE policy decisions. Below you will find summaries of recent reports as well as links to download the complete PDF reports.
Tennessee launched a new, statewide system of educator evaluation in 2011. In each of the years since, there have been hundreds of thousands of observations and conversations about teaching practice. The teacher observations that represent the primary component of teacher evaluation scores have the potential to provide valuable information about teachers’ instructional strengths and areas in need of improvement—information that can be used both by district personnel and by the teachers being evaluated. But the value of observers’ ratings of teaching practice depends in large part on the strength of the feedback that the ratings provide.
Currently, the most common way of assessing the quality of observers is to examine the alignment between the ratings teachers are given in observations and value-added ratings that measure teachers’ impact on student test scores. This can be a highly useful metric, and it is one that we make use of in Tennessee when we think about the overall accuracy of observation ratings. The alignment metric, however, focuses more on the validity of the rating than the quality of the observer feedback.
This report describes an analysis by the Tennessee Department of Education’s Office of Research and Policy that proposes a second quantitative metric for assessing the practicality and usefulness of the ratings that teacher evaluators are giving to the teachers they observe by looking at the range of ratings given. We introduce the idea of the “non-differentiating observer,” or the observer whose ratings do not usefully distinguish between teachers’ relative strengths and weaknesses. This metric highlights the need for evaluations to be useful for teachers as a tool for their own improvement, and it allows us to assess evaluator performance in an ongoing way during the year.
In the following pages, we identify the number of non-differentiating observers, where they are located, and whether any particular characteristics predict whether an observer will be nondifferentiating. While non-differentiating observers represent only a small proportion of the overall observer pool, we are able to identify a group that is clearly distinguishable within the data, and we propose several possible supports and interventions that might reduce the problem. It is our hope that this paper will both contribute to the process of continuous improvement to the teacher evaluation system in Tennessee and help other states, districts, and schools think critically about their own processes for assessing observer quality.
- Observers whose ratings do not provide teachers with a range of feedback on strengths and weaknesses are failing to provide the kind of usable information that might lead to instructional improvement.
- In Tennessee, “non-differentiating observers” make up a small but meaningful proportion of the total evaluator pool.
- Non-differentiating observers are not clustered in particular areas but are scattered throughout the state and across all types of observers.
- The real-time observation data collected by the Tennessee evaluation system can allow districts and the state to identify non-differentiating observers during the year and to take steps to ensure that teachers are receiving meaningful feedback.
- Non-differentiation is one of several indicators of low observation quality, and non-differentiating observers are not necessarily the same observers who fail to achieve reasonable alignment between observation and value-added ratings.
Starting in 2014-15, districts across the state must implement a Response to Instruction and Intervention (RTI²) plan. RTI² is an instructional framework which provides ongoing monitoring of student performance and progress through the use of universal student screeners and interventions targeted at students’ identified problem areas. The framework is meant to identify and reduce student skill deficits and to provide a consistent method for diagnosing special education candidates.
For some districts and schools, the RTI² framework is entirely new. Others have used some version of the program, often in select grades or subjects, for several years. The variation in RTI² preparedness across the state suggests the need for state personnel to provide a variety of different support strategies to insure strong implementation statewide.
This report attempts to survey the spectrum of district and school readiness through a series of case studies in order to identify the kinds of targeted support that might prove most useful. To gain
perspective on the implementation of RTI², we interviewed district and school leaders from 14 schools in seven districts across the state.
- Nearly all of the district and school administrators that we spoke with demonstrated a remarkably strong knowledge of the state’s RTI² framework, suggesting that state communication and trainings have been quite successful in raising awareness of the new requirements.
- Scheduling has proved to be a major difficulty for most schools, and administrators say they are unable to find enough time in the school day to adhere to state RTI² scheduling guidelines around uninterrupted core instruction and intervention time and still retain time for activities such as Art and Physical Education.
- Nearly all administrators feel that RTI² has placed additional strain on already thin resources, and they identify challenges in financing the initial system, building the human capital to execute the program, and finding the physical space for the required small group interventions.
- Many schools still are struggling to successfully blend the silos of general and special education in order to create the collaboration that feels necessary for strong RTI² implementation.
- Most districts are still focused on identifying screeners and progress monitors and few have reached the point where they are thinking deeply about the interventions that will take place once deficits have been identified.
This report examines the extent to which teacher retention rates in Tennessee schools differ according to teachers’ effectiveness and the ways this information might inform strategic retention efforts at the state and district levels. We find promising evidence that improvements in certain working conditions have the potential to improve the retention rates of highly effective teachers.
- Teachers who earn higher overall teacher evaluation scores tend to be retained at slightly higher rates than teachers who earn lower overall teacher evaluation scores, although the differences in these rates are not particularly large.
- Early career teachers are slightly less likely to be retained than other teachers. Highly effective early career teachers tend to be retained at slightly higher rates than other early career teachers.
- Highly effective minority teachers are considerably more likely to leave Tennessee public schools than other highly effective teachers.
- There is substantial variation across districts in overall retention rates, retention rates of teachers earning high evaluation scores, and the degree to highly effective teachers are retained at a higher rate than other teachers.
- School conditions such as effective time use and functional teacher evaluation were significantly related to retention rates of highly effective teachers. As a result, strategies aimed at improving these factors have the potential to improve the retention of these teachers.
To better understand writing instruction in Tennessee, we examined student responses to survey questions administered with the 2013 Writing Assessment. The questions asked about writing practices and practice using computers for writing. We also looked at 2012-13 school-level technology survey data to determine computer resources available to educators and students.
- State-level findings showed that students who reported more frequently (a) writing in non-ELA courses, (b) working with peers to discuss writing, and (c) making notes or an outline before writing had higher Writing Assessment scores.
- The survey data also revealed that while some students in Tennessee are engaging in these practices, many others are not.
- Furthermore, economically disadvantaged students reported significantly less practice using computers for writing, although they have slightly more computers per student in their schools.
In 2013, Tennessee counted nearly 7,000 students in the senior cohort whose academic skills when they entered high school suggested they were on track to earn college credits through Advanced Placement (AP) exams. Yet just over half of these students actually graduated with an AP credit, and less than a third of the economically disadvantaged students made the grade. What happened?
The following report details real issues schools encounter when moving academically prepared students along the AP pipeline, from access to success. This is not a story of student performance declines. Rather, it is one of missed opportunities for school-level interventions, in the form of additional course offerings, more targeted student counseling, or greater financial support for AP students. By using student-level data to highlight particular issues that schools face, we hope to create a framework that allows schools and districts to design targeted solutions to their individual challenges. While AP is only one of several options for allowing high school students to earn post-secondary credit, the framework applied to AP in this report could serve equally well for considering other options such as dual credit, dual enrollment, or International Baccalaureate programs.
- Tennessee lags far behind the nation on Advanced Placement (AP) testing rates and early college credits earned by passing AP tests.
- Eighth grade Tennessee Comprehensive Program (TCAP) scores are a strong predictor of student success on AP exams.
- In order to increase Advanced Placement success we need to think critically about the availability of AP opportunity.
- In some schools, increasing opportunity means training teachers and offering a wider array of AP courses. In others, it means ensuring that qualified students from all backgrounds enroll in available courses. And in others it means pushing students who have taken AP classes to sit for the actual exam and the chance to earn college credit.
- Targeted programs and interventions have the potential to change patterns in specific schools and launch students and the state on the pathway toward greater AP success.
During the spring and summer of 2012, the Tennessee Department of Education trained about 200 Core Coaches who then went on to facilitate summer trainings for thousands of the state’s third through eighth grade math teachers. The following summer, the training sessions reached nearly 30,000 teachers across the state, covering math, English language arts, and literacy in science, social studies, and career and technical education.
The Tennessee Core Coach training model was designed to develop a network of teachers with a deep content and pedagogical knowledge of the Tennessee Academic Standards who could pass the knowledge on to their peers during formal training sessions and informal interactions throughout the year. Coaches were Tennessee teachers selected via a competitive application and interview process. Coaches received eight days of intensive grade-level training provided by the Institute for Learning at the University of Pittsburgh, engaging with the material first as learners and then as teacher trainers. Coaches then delivered training to participants at three-day, grade-level workshops held throughout the summer.
This memo provides the first analysis of the effectiveness of the TNCore training on teacher practice and student achievement. The findings suggest that attendance at the summer training sessions made a significant difference to student achievement and teacher effectiveness, as measured by both observer ratings and value-added scores based on statewide TCAP math tests. These results remain consistent even as we apply evaluative approaches that use multiple years of teacher data and control for prior scores and school effects, suggesting that these results are not biased by participant selection.
- We consistently find positive and significant effects of the TNCore math training on participants’ instructional practice and on their effectiveness at raising student test scores. These results remain consistent using methods that control for previous year scores, school-level inputs, and for the fixed characteristics of teachers.
- Participants’ gains on observation scores were equivalent to about half of the gains made by the average teacher between the first and second year of teaching.
- The gains in instructional practice ratings were largest for the practices emphasized in the training sessions, including skills such as questioning, providing academic feedback, and teaching problem-solving techniques.
- Participants’ gains in effectiveness as measured by the Tennessee Value-Added Assessment System (TVAAS) translate into the equivalent of approximately one extra week of learning for each of their students than we would have expected had they not attended the training sessions.
- Participants who had a Core Coach working at their school made significantly greater estimated increases in questioning practices compared to participants without this support.
- For Core Coaches, we find some evidence that the coaching process was associated with improvements in their own classroom teaching; however, we are uncertain whether these improvements can be attributed to their role as coaches.
Across Tennessee, student test scores have risen steadily in every subject since tests were revised in 2009-10, so that nearly 91,000 additional students are performing at grade level in math and nearly 52,000 additional students are performing at grade level in science.
This report focuses on some of the schools that are making this possible. Each year, Tennessee recognizes 10 percent of its schools for overall excellence. The list includes the schools that have made the greatest gains from the previous year and the schools that achieved at the highest levels of absolute performance across the state. In the following pages, we shine a light on these schools, sharing some of their stories and practices as a source of inspiration and learning.
- 169 out of 1,668 schools were recognized for Reward status.
- Reward Schools represent 52 districts and span all school sizes and types.
- 19 percent of Reward Schools serve predominantly black, Hispanic, or Native American students.
- Several years ago, Coffee County High School was deemed to be failing under the previous No Child Left Behind regulations. Since that designation, Coffee County High School performance has grown at record levels, and the school has been recognized as a Reward School two years in a row.
- In 2011, Lowrance Elementary School had the lowest math test scores of all elementary schools in Shelby County. Only 20 percent of students were proficient or advanced. This year, almost 40 percent of students reached grade level in math. Like Coffee County High School, Lowrance is a Reward School for the second year.
Room for Improvement
- Our neediest students are still underrepresented in our best schools. On average, our Reward Schools have fewer minority and economically disadvantaged students than the state average.
- Our best schools achieve extraordinary results on our state tests, but there is still significant work to be done to ensure that all students across the state graduate high school ready for success in college and career.
Tennessee has undertaken several wide-ranging reforms aimed at educator practice and effectiveness over the past several years, including the launch of a new teacher evaluation system, modification of teacher tenure policy, and initial implementation of Tennessee Academic Standards. This report from the Tennessee Department of Education (TDOE) investigates patterns of teacher retirement in the years surrounding these policy changes. Our analysis is concerned not only with the overall levels of teacher retirement in the state but also with the relative effectiveness of those who stay and leave the teacher workforce through retirement.
We find that retirement rates have risen since 2009, corresponding with a loss of an additional one percent of the teacher workforce or approximately 1,000 teachers per year. The rise in retirement has been steady since just prior to the onset of the reforms under Race to the Top, and has continued through the ensuing years. At the same time, there is evidence that current patterns of retirement are likely to strengthen the overall quality of the teacher workforce. First, among those eligible for retirement, teachers who choose to retire tend to be less effective than those that remain in the classroom. Moreover, retirement rates are becoming increasingly differentiated by teacher effectiveness, with the more effective teachers choosing to stay longer in the classroom and the less effective teachers choosing to leave at higher rates.
- Between 2008 and 2012, the rate of teacher retirement from the workforce increased from 2.0 percent to 3.5 percent.
- Retiring teachers consistently rate lower in effectiveness than those teachers eligible for retirement that choose to remain in the classroom; similarly, retiring teachers consistently rate lower than all remaining teachers regardless of eligibility for retirement.
- In 2010, teachers at different ends of the effectiveness spectrum retired at nearly identical rates. Since that year, the rates have diverged to the point where the state’s most effective teachers