Project Impact
Design
The goal of the ARTS ACHIEVE study was to test the hypothesis that arts assessment data, coupled with professional development and technology integration, would improve arts teacher instructional practice and thereby improve student learning in the arts. To test this hypothesis, an experimental study was designed, whereby schools were randomly assigned to the treatment or status-quo control condition. This design allows for comparison between teachers and students who participate in the program and those who do not.
METHODS
Study Sample
New York City public schools that met basic eligibility requirements were recruited and then randomly selected to participate. The selected schools were then randomly assigned to either the treatment or control group. Schools were randomly assigned using a stratified randomized process. The strata were based on arts discipline (dance, music, theater, and visual arts) and school level (elementary, middle, and high school). The treatment was mainly targeted towards the arts teacher in the school’s designated arts discipline.
Data Collection
Data sources for ARTS ACHIEVE consisted of program implementation documentation, surveys and focus groups/interviews with arts teachers, and data from the Arts Performance Assessments. Additionally, secondary data were collected from the New York City Department of Education, including students’ background characteristics and English language arts achievement.
Arts teacher surveys were developed by the evaluator and program staff to collect information on arts teachers’ perceptions of their knowledge and skills and their self-reported instructional practices. The surveys were administered to participating arts teachers in the treatment and control schools prior to the start of implementation (baseline) and at the end of (post) of each school year. The surveys used Likert-scale and open-ended questions.
The Arts Performance Assessments were developed and piloted in the year prior to ARTS ACHIEVE implementation (2010-2011) and were used to measure students’ arts achievement. The New York City arts directors in the four arts disciplines led Curriculum Assessment Development (CAD) teams to design the assessments. The CAD teams were comprised of school-based arts teachers, staff from the participating arts organization, and New York City Department of Education experts in tests and measurement. A total of 12 assessments were developed (one in each arts discipline and school level). The assessments are aligned with the New York City Blueprint for Teaching and Learning in the Arts: PreK-12, as well as the Common Core Capacities in English Language Arts.
Data Analysis
Multiple regression analyses were conducted to measure the impact of the ARTS ACHIEVE project on arts teachers’ instructional practice, as measured by the arts teacher surveys. Three-level hierarchical linear modeling (HLM) was conducted to measure the impact of the ARTS ACHIEVE project on students’ arts achievement. Potential confounding factors that have a relationship with the treatment or outcome are included, in both the multiple regression as well as the HLM analyses, to reduce threats to the study’s internal validity.
RESULTS
Impact on Teacher Instructional Practice
To measure whether change occurred in teacher practice, arts teachers completed annual pre/post surveys, through which they self-reported the frequency with which they used formative assessment strategies (i.e., teacher descriptive feedback, peer feedback, student self-reflection, and student self-assessment), taught the strands of the Blueprint (e.g., developing arts-based literacy, making connections through the art form), and used technology (e.g., iPads, Smart board) in instruction. Rating scales ranged from 1 (lowest use) to 6 (highest use). An average across items was calculated to create a composite measuring arts teachers’ instructional practice.
Multiple regression analyses of the instructional practice data indicate that there was a statistically significant effect of ARTS ACHIEVE on arts teachers’ instructional practice in both Years One and Two of implementation. In Year Three of implementation, although the arts teachers in the treatment schools made greater gains than the arts teachers in the control schools, these gains were not statistically significant. The analyses controlled for teacher background characteristics and also adjusted for lack of baseline equivalence between the treatment and control groups. The Year 1 effect size was 0.53, the Year 2 effect size was 0.59, and the Year 3 effect size was 0.27.
Impact on Student Arts Achievement
In each year of implementation, students in the treatment schools made greater gains in their arts achievement than participating students in the control schools. The results of HLM analyses indicated that participating students in the treatment schools made significantly greater gains in their arts achievement than participating students in the control schools, after controlling for student, teacher, and school background characteristics in Years One and Two of implementation. The gains between the students in the treatment and control schools were not statistically significant in the third year of implementation. The Year 1 effect size was 0.28, the Year 2 effect size was 0.20, and the Year 3 effect size was 0.09.
*p<.05
While the first two years of ARTS ACHIEVE show extremely strong results, the data indicate that the effect in Year 3 was not as strong. There are many possible explanations for the lack of significant findings in the final year of implementation. For example, it may be due to the greater amount of time that teachers and facilitators spent on documenting the ARTS ACHIEVE work in the final year as compared to the first two years. Though these efforts were extremely useful for replication efforts, several arts teachers and facilitators reported that it took away from implementation efforts. Additionally, it may be that the lack of findings regarding teachers’ instructional practice is due to low response rates. Survey data collection from the participating treatment and control teachers was often challenging, becoming more so over time.
See the full report for more detailed findings.
