Study Overview

• Rationale for the Study
• Research Questions and Evaluation Design
• Timeline and Data Collection Plan
• Confidentiality

Rationale for the Study

Early in 2005, the U.S. Department of Education's Institute of Education Sciences (IES) worked with Mathematica Policy Research, Inc., to convene a panel of experts in mathematics, mathematics instruction, and evaluation design to (1) discuss issues that confront a study of the effectiveness of mathematics curricula, and (2) recommend how the study could address those issues. This panel helped shape the study by identifying the early elementary grades as the most important level for the evaluation because, even before they enter elementary school, disadvantaged children fall behind their more advantaged peers in basic competencies such as number line ordering and magnitude comparison. Research also shows that learning these basic skills in early elementary grades is essential to learning fractions in later elementary grades.

The expert panel also recommended an evaluation that compares different approaches to teaching early elementary mathematics, and advised that many mathematics curricula have been developed recently that are being widely implemented without evidence of their effectiveness. In addition, several basal math textbooks have been widely used over the years—also without evidence of effectiveness.

Back to top

 

Research Questions and Evaluation Design

In October 2005, IES began working with Mathematica to conduct the Evaluation of Mathematics Curricula. The evaluation seeks to answer three questions:

• What are the relative effects of different early elementary math curricula on student math achievement in disadvantaged schools?
• Under what conditions is each math curriculum most effective?
• Which math curricula result in a sustained impact on student achievement? 

 

Experimental methods are being used to answer the research questions listed above. In particular, the evaluation is based on a school-level random assignment design, in which participating elementary schools in each participating district are randomly assigned to the curricula included in the study. Consider, for example, a district that has eight elementary schools participating in the study. The study team randomly selected two schools to implement curriculum A, two schools to implement curriculum B, and so on. In each school, teachers at the target grade levels receive training and both teacher and student materials free of charge for the curriculum assigned to their school. Relative effects of the curricula are estimated by comparing average math achievement of students in the various curriculum groups.

The study is designed to detect an effect size as small as 0.20 between any pair of curricula. To detect the study's target effect size, the study recruited 12 districts and 111 schools to participate. Districts and schools were selected for their suitability and willingness to comply with the study design. In general, participating schools serve a large number of low-achieving students or students in poverty. A key requirement for participation is that district and school staff were willing to permit the study determine through random assignment the curriculum their schools will use. Geographic representation and urban or rural location also were factors in identifying districts and schools.

Schools in the study used their assigned math curriculum in the first grade during the 2006-2007 school year, the first and second grades during the 2007-2008 school year, and the first, second, and third grades during the 2008-2009 school year.

 

Back to top

 

Timeline and Data Collection Plan

The evaluation began in October 2005 and is a five-year study. The first year involved several activities that set up the study. Among these activities were (1) selecting the curricula, (2) recruiting districts and schools, (3) developing data collection forms, and (4) finalizing the evaluation design. Implementation of the curricula and data collection took place in first grade during the 2006-2007 school year, in first and second grades during the 2007-2008 school year, and the first, second, and third grades during the 2008-2009 school year. A report based on the 2006-2007 school year is available. A report based on both the 2006-2007 and 2007-2008 school years is expected in Fall 2009. A report based on the third year of implementation is expected in September 2010.

The study includes several complementary data collection efforts that support answers to the study's research questions.

• Assessment of Student Achievement. The study team is measuring student math achievement using the assessment developed for the National Center for Education Statistics’ Early Childhood Longitudinal Study-Kindergarten Class of 1998-99 (ECLS-K). The assessment is individually administered, standardized, and nationally normed.
• Assessment of Teacher Knowledge of Math Content and Pedagogy. Teacher math content/pedagogical knowledge was assessed at the initial teacher training sessions before the curricula were introduced, using an assessment developed by researchers at the University of Michigan. 
• Teacher Surveys. Two surveys are administered to teachers. The first survey is administered in the fall and focuses on teacher background information, classroom characteristics, curriculum training provided by the publishers up to that point, and math instruction approaches used before joining the study. The second survey is administered in the spring and gathers information on follow-up training provided by the publishers, usage of the assigned curriculum and any other math curricula, and math instructional practice used during the year. 
• Class Rosters. The study will collect rosters for each classroom to identify the student sample. Student demographic information will be requested as part of the roster collection, so the study can define subgroups for investigating whether curricula effects differ for subgroups.    
• Classroom Observations. Classrooms in the study are observed once during the implementation years. Information collected during the visits will be used to contrast teaching practices and student activities in the classrooms using the various curricula. 
• Teacher Training Observations. The study observed the teacher trainings provided by the publishers before the start of the school year. Observations will be used to characterize the initial trainings and their fidelity to publisher-specified standards.

 

Confidentiality

All information collected by the study is strictly confidential and carefully guarded to the full extent allowed by law. The study's results will be reported only for groups of students, such as “70 percent of first graders achieved mastery of rudimentary mathematics.”

Back to top