Contact our sales team

Thank you!

Sed posuere consectetur est at lobortis. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum.

Close

Please confirm your state so we can curate content for your location.

Confirm

Amplify Science is associated with better grade 5 science outcomes in Washington

Insight

By Leah Walker, PhD, Executive Director Assessment and Reporting at Amplify

Introduction

This paper describes a study of science outcomes for schools using Amplify Science based on analysis of the Washington Comprehensive Assessment System Grade 5 Science test. The study was designed as a test of the product’s theory of action: schools using Amplify Science should keep pace with or even outperform schools using other programs. Specifically, a multiple regression analysis was conducted to investigate the relationship between a school’s use of Amplify Science and science performance, while controlling for selection bias using previous scores and school-level demographic variables.

Amplify Science

Amplify Science for elementary school is a core science curriculum program in grades K–5. Teachers using Amplify Science have access to digital tools and downloadable materials to be used in the classroom. Teachers also receive kits to support students’ scientific investigations in the classroom and beyond. Amplify Science was designed to keep students interested and engaged while covering key topics in science. The four units that were specifically designed for 5th-grade students cover: Patterns of Earth and Sky, Modeling Matter, The Earth System, and Ecosystem Restoration.

Figure 1. Amplify Science 5th-grade units

Each unit contains a teacher guide that explains what the unit covers, why these topics are important for student success in science, and how Amplify Science organizes these topics in a developmental progression. These guides make the logic model of each unit available to teachers in practical language. This makes it more likely that teachers will implement lessons according to the recommended sequence and be able to track student outcomes in real time against classroom observations and assessments. For example, the excerpt from the Patterns of Earth and Sky Teacher guide explains how the models, simulations, and readings in the unit facilitate students growth in spatial reasoning ability and the ability to explain phenomena (Lawrence Hall of Science, 2018):

The spatial reasoning involved in understanding many space science ideas is challenging. In this unit, it involves understanding the position of stars in relation to Earth and the sun, as well as figuring out how Earth’s spin and orbit cause us to see different things across a day and across a year. Over the course of the unit, students have repeated opportunities to investigate these patterns, through multiple models. Through the use of these models, students also begin to develop a sense of the large distances and scale of objects in the universe. Access to these ideas through a series of kinesthetic models, physical models, a computer model, and text enables students to have the collection of experiences that are necessary for students to begin to own these ideas. Being able to explain the illustrations on an ancient artifact that shows changes in the sky over time and to figure out what might be on the missing piece is an intriguing and complex task. It’s one that requires knowledge of Earth’s daily and yearly movement as well as an understanding of gravity. As such, the problem provides a compelling series of real world phenomena (as captured in the illustrations on the artifact) for students to figure out and explain. In addition, the task of explaining the artifact unifies a set of related and important space science ideas, often taught separately.

Within each unit, teachers can access subunits that contain a series of lessons. The lessons provide all of the media, materials, and guidance necessary to facilitate teacher-led and student-to-student discussions, perform digital simulations of processes, set up hands-on experiments with materials shipped in the kits that accompany Amplify Science, perform student assessment, and have students practice writing and reading about science.

Figure 2. The variety of materials contained in the teacher lesson guides

Present study

Amplify Science was designed to improve student outcomes by relying on a strong theory of action for each unit. This study investigates whether use of Amplify Science had a positive impact on student scores. Specifically, this study seeks to answer the following questions:

  1. Is use of Amplify Science associated with stronger science performance than use of other programs when controlling for selection bias?
  2. If there is a difference in learning outcomes for Amplify Science users, is this difference statistically significant?

Method

5th-grade science outcomes from 2016–2017 were collected from Washington schools that would and would not be using Amplify Science during the 2017–2018 academic year. These outcomes were used as a pretest measure. During the following academic year (2017–2018), teachers in the Amplify Science schools were then given access to the four Amplify Science units for 5th grade. 5th-grade science outcomes from all of the schools were then collected again in 2017–2018 and used to study the relationship between science performance and use of Amplify Science.

Participants

The treatment group was defined as elementary schools that had Amplify Science during the 2017–2018 academic year and had access to all four units for 5th grade. Schools that also had access to Amplify’s middle school science curriculum were excluded from the treatment group because this product allows both teachers and students to access the digital platform independently. By excluding users of both Amplify elementary and Amplify middle school curriculum products, the relationship between use of Amplify’s elementary school science curriculum and science performance could be investigated separately. Thus, for the purpose of this study, an elementary school was considered any public school that included grade 5 but did not also include grades 6, 7, or 8. The grade spans for elementary schools included in this study, in order of frequency, were: K–5, PK–5, 3–5, 1–5, 2–5 and 4–5, with K–5 and PK–5 schools making up 97% of the sample. The control group was defined as elementary schools that did not have Amplify Science during the 2017–2018 academic year.

Table 1. Number of treatment and control units

Instruments

Performance on the Washington Comprehensive Assessment System (WCAS) science test for 5th grade was used as both a pretest and posttest measure. Scores from the 2016–2017 academic year were used as the pretest, and scores from the 2017–2018 academic year, after schools had access to the Amplify Science units for at least 8 months, were used as a posttest. The WCAS is a large-scale standardized assessment developed by the State of Washington to place students into one of four Achievement Levels that outline which types of academic tasks a student can master and whether they have met the standards adopted by the State. Students who attain Levels 3 and 4 are considered to have met the standard, while students in Levels 1 and 2 have not. More information on the Grade 5 WCAS Achievement Levels is available through the Washington Office of the Superintendent of Education and Instruction. The percent of students in the school meeting the standard according to the WCAS achievement levels was used as the pretest and posttest metric.

School-level demographic variables were used as additional controls to provide for adequate comparisons. In model 1, these demographic variables include the % of students who were male, the % of students who were white, and the % of students who were transitional bilingual ELLs. In model 2, the same demographics were included, in addition to the % of students receiving free or reduced price lunch. Both models were considered because, although the % of students receiving free or reduced price meals is generally considered to be an accurate representation of low socio-economic status, some schools and districts have instituted policies of blanket enrollment in the free and reduced price program for all students.

Results

5th-grade science performance for the treatment and control groups was compared to determine its relationship to the use of Amplify Science. To account for selection bias, model 1 included statistical controls for previous year performance, gender, race/ethnicity, and English language learner status, and model 2 included the same with the addition of a control for low socio-economic status. After controlling for these variables, model 1 shows that Amplify Science was associated with a statistically significant positive effect. Specifically, Amplify Science is associated with an average +3.47 point difference in the % of students meeting the standards for 5th-grade science, as measured by the WCAS Science Test in 2017–2018, which is a statistically significant result. However, model 2, which also controls for differences in socio-economic status, showed an average +2.27 point difference in the % of students meeting the standards for 5th-grade science, or an effect size of +0.11, a result that was not statistically significant.

Figure 3. Comparison of science test results excluding control for SES and including control

To describe these differences in terms of standard deviation (SD) units, the posttest outcome, along with all of the continuous predictor variables (previous year performance, percent male, percent white, percent transitional bilingual) were standardized, and the model was re-estimated. This second set of estimates shows that Amplify Science is associated with an average +0.19 SD difference in the % of students meeting the standards for 5th-grade science under model 1 and a +0.12 SD difference in the % of students meeting the standards for 5th-grade science under model 2. Effect sizes of both +0.19 and +0.12 are classified as small (Cohen, 1998) or medium (Kraft, 2018). These effect sizes can be translated into differences in percentiles as well. Model 1 results say that the differences are equivalent to the 50th percentile vs. the 58th percentile, and model 2 results say that the differences are equivalent to the 50th vs. the 54th percentile, a change of 4 percentile points, which is not significant.

Table 2. Regression model showing how performance is related to Amplify Science

To help visualize the effect described by these estimates, the expected average growth trajectories for each group have been plotted, starting with an average of 64.70% of students meeting the standard for schools in both groups in Year 1. Figure 4 shows that, on average, Amplify schools would have 69.51% of students meeting the standard in year 2, whereas other schools would also show growth, but have 66.04% of students meeting the standard in year 2, a difference of 3.47 percentage points in year 2 outcomes. Figure 4 also illustrates that this difference is significant and enough to distinguish Amplify schools outside of the margin of error.

Discussion

The purpose of this study was to investigate the following questions:

  1. Is use of Amplify Science associated with stronger science performance than use of other programs when controlling for selection bias?
  2. If there is a difference in learning outcomes for Amplify Science users, is this difference statistically significant?

Two models were fit to describe performance in science, controlling for previous performance and demographic variables. Model 1 included all of the same controls as model 2, except for the free and reduced price lunch variable. Both models were estimated and presented because it is not clear whether the free and reduced price lunch provides an accurate indicator of low socio-economic status in all cases, especially for schools who enroll all students in the program. However, taking into account that the free and reduced price lunch variable was a significant predictor and because another indicator of socioeconomic status is not available, only the results from model 2 are discussed further below.  

The results show that use of Amplify Science is associated with an average 2.27 point gain in the % of students meeting the grade 5 science standards (effect size +0.11), a result which was positive but not statistically significant.

Over the next few years, growth in the number of schools using Amplify Science is expected, and additional PD and supports are planned. In the future, similar data could again be collected and analyzed. The evidence from the current study suggests that the results from these larger and more supported implementations may show positive significant outcomes for schools and students.  

References

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

Institute of Education Sciences (2017). What Works Clearing House Standards Version 4.0. Retrieved from https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_standards_handbook_v4.pdf

Kraft, M.A. (2018). Interpreting Effect Sizes of Educational Interventions [Brown University Working Paper]. Retrieved from https://scholar.harvard.edu/files/mkraft/files/kraft_2018_interpreting_effect_sizes.pdf

Lawrence Hall of Science (2018). Patterns of Earth and Sky: Analyzing Stars on Ancient Artifacts – Teachers Guide. New York, NY: Amplify Education, Inc.

United States Department of Education (2016). Non-Regulatory Guidance: Using Evidence to Strengthen Education Investments. Retrieved from https://ed.gov/policy/elsec/leg/essa/guidanceuseseinvestment.pdf

About the author

Leah Walker, PhD Executive Director Assessment and Reporting at Amplify

More Insights

Insight

#AmplifyYourSummer with free books!

Make any read a beach read this summer with free use of the Amplify Library! Sign up now for 90 days you’ll have access to more than 650 titles, from memoir to history to biography to fantasy: the books kids love now and...

Read more

Insight

How do we make assessment more like a Fitbit? Part II

In my last post, I compared my beloved Fitbit watch to the ideal assessment system. My Fitbit does formative assessment right—it runs in the background and shows me my progress toward my daily activity goals in real time...

Read more
See all insights