nav HOME

nav arrow INFORMATION FOR

navigation arrow PUBLICATIONS

nav arrow CONTACT US

 

Study Overview

Rationale for the Study

Early in 2005, the U.S. Department of Education's Institute of Education Sciences (IES) worked with Mathematica Policy Research, Inc. (MPR) to convene a panel of experts in mathematics, mathematics instruction, and evaluation design to (1) discuss issues that confront a study of the effectiveness of mathematics curricula, and (2) recommend how the study could address those issues. This panel helped shape the study by identifying the early elementary grades as the most important level for the evaluation, because research shows that in basic competencies (such as number line ordering and magnitude comparison) disadvantaged children fall behind their more advantaged peers even before entering elementary school. Research also shows that learning these basic skills in early elementary grades is essential to learning fractions in later elementary grades.

The expert panel also recommended an evaluation that compares different approaches to teaching early elementary mathematics, and advised that many mathematics curricula have been developed recently that are being widely implemented without evidence of their effectiveness. In addition, several basal math textbooks have been widely used over the years—also without evidence of effectiveness.

Back to top

 

Research Questions and Evaluation Design

In October 2005, IES began working with MPR to conduct the Evaluation of Mathematics Curricula. The evaluation seeks to answer two questions:

  1. What are the relative effects of different early elementary math curricula on student math achievement in disadvantaged schools?
  2. Under what conditions is each math curriculum most effective?

Experimental evaluation methods will be used to answer the research questions. In particular, the study will use a school-level random assignment design to answer the question of effectiveness of math curricula. This design calls for randomly assigning schools in each participating district to the selected curricula, and comparing math achievement of students in those schools.

Consider, for example, a district that has four elementary schools willing to participate in the study. The study will randomly select the school that will be assigned to implement curriculum A, the school that will be assigned to implement curriculum B, and so on. In each school, teachers in the early grades will use the curriculum assigned to their school. Relative effects of the curricula will be estimated by comparing average math achievement of students in the target grade levels in the various schools. For example, the relative effectiveness of curriculum A versus curriculum B will be estimated as the difference in average achievement between early elementary students in the school assigned to curriculum A and those in the school assigned to curriculum B.

The study will evaluate four curricula, and the goal is to detect an effect size as small as 0.20 between any pair of curricula. To detect the study's target effect size, the study will recruit about 15 districts and 100 schools to participate. Districts and schools will be selected for their suitability and willingness to implement the curricula rather than through a formal sampling process. In general, the schools identified will serve a large number of low-achieving students or students in poverty. A key requirement for participation is that district and school staff are willing to have the study determine through random assignment the curriculum their schools will use. Geographic representation and urban or rural location also will be factors in identifying districts and schools.

Schools in the study will use the math curricula in the first grade during the 2006-2007 school year and in the first and second grades during the 2007-2008 school year. The study also contains an option to implement the curricula in the third grade during the 2008-2009 school year.

 

Back to top

 

Timeline and Data Collection Plan

The evaluation began in October 2005 and is a four-year study, with an optional fifth year. The first year involves several activities that set up the study. Among these activities are (1) selecting the curricula that will be included, (2) recruiting districts and schools, (3) developing data collection forms, and (4) finalizing the evaluation design. Implementation of the curricula and data collection will take place in first grade during the 2006-2007 school year and in first and second grades during the 2007-2008 school year. A report based on the two grade levels is expected by September 2009. Another year of curricula implementation and data collection may occur during the 2008-2009 school year, and a report based on this school year would be expected by September 2010.

The study includes several complementary data collection efforts that support answers to the study's research questions.

  • Teacher Training Observations. The study will observe the teacher trainings provided by the publishers before the start of the school year. Observations will be used to characterize the trainings and their fidelity to publisher-specified standards.
  • Assessment of Teacher Knowledge of Math Content and Pedagogy. The study will assess teacher math content and pedagogical knowledge at the teacher training sessions before the curricula are introduced, to examine the relationship between teacher knowledge and curricula effects. An existing assessment that has been shown to be valid and reliable will be used.
  • Teacher Surveys. Two teacher surveys will be administered. The first (baseline) survey will be administered in the fall and will focus on curriculum training, math instruction and strategies, classroom characteristics, and teacher background characteristics. The second (implementation) survey will be administered in the spring and will gather information about the types of curriculum support teachers received during the school year, curriculum usage, math instructional practice, and general perceptions about the curriculum used.
  • Classroom Observations. Classrooms in the study will be observed once during the implementation years. Information collected during the visits will be used to contrast teaching practices and student activities in the classrooms using the various curricula.
  • Assessment of Student Achievement. The study will assess math achievement of the students in the teachers' classrooms participating in the study at the beginning and end of the school years. The assessment is an individually administered test that is standardized and nationally normed.

Confidentiality

All information collected by the study is strictly confidential and carefully guarded to the full extent allowed by law. The study's results will be reported only for groups of students, such as “70 percent of first graders achieved mastery of rudimentary mathematics.”

Back to top