Instructional sensitivity: Why teachers intuitively know standardized assessments aren't good measures of student learning

The annual Primary Sources report by Scholastic and the Bill & Melinda Gates Foundation examines teacher attitudes on a number of topics.  Recent headlines from this year's report focused on teacher perceptions of standardized testing.  For some good reasons worth discussing, a majority of teachers do not find standardized tests to be good measures of student learning or teaching effectiveness.  But the survey also revealed encouraging developments in teacher perceptions of formative and classroom-based assessments and the value of using the data from such measures in evaluating teacher performance.

Read the Education Week/Teacher magazine report on the survey here.

The national survey of more than 10,000 teachers found that only about a quarter of teachers consider standardized tests to be good measures of student learning.  Nearly half of all teachers say that students themselves don't take the tests very seriously.

This is a valid point, but there are better arguments for why standardized assessments have limited value for accountability and teacher evaluation purposes.  W. James Popham, author of Transformative Assessment, Everything School Leaders Need to Know About Assessment and other books, says that for a test to be used formatively, either to inform instruction or make judgments about the quality of instruction, assessments must be "instructionally sensitive."  In other words,

An instructionally sensitive accountability test would be one that would include many items that uninstructed students would tend to answer incorrectly and instructed students would tend to answer correcty.

Makes sense, right?  But for a test to meet this definition, Popham says it must have certain characteristics:

• First, it must be based on a modest number of important curricular targets.  In other words, the test can't attempt to measure too many different standards.  This is why assessment experts from Marzano to Schmoker to Reeves advocate that schools condense and prioritize their curriculum to focus on power standards that have leverage and endurance, and make students ready for the next grade level.  If a test tries to measure too many different standards, the number of items per standard will be too small to offer a valid measure of whether students have actually mastered the knowledge and skill described by the standard.
• Second, instructionally sensitive tests are based on learning targets that are clearly defined.  Teachers and students alike must be absolutely clear about the specific knowledge and skill that each of these prioritized standards is meant to describe, so that test items can match those skills accurately.  If a test tries to measure a standard that is too broad and vague, teachers and students are unlikely to be prepared for the various ways the items associated with that standard might be presented.

Third, performance reports for instructionally sensitive tests must generate data that shows exactly how each student performed relative to each curricular aim.  This requirement breaks test results down so that a student (and his teacher) can see exactly what the student knows and is able to do relative to the learning targets the test was meant to measure.  If a test does not give this per-student/per-standard feedback, it is extremely difficult to determine what a student has actually learned or the extent to which a teacher has successfully taught the material.

Finally, each item of the assessment must be carefully constructed to ensure that a student's cultural/socio-economic background or natural aptitude can't give him an advantage in answering the question correctly.  If he could, then the test item doesn't measure the teacher's impact on student learning, but rather other factors separate from what has actually happened in the classroom. 

Most standardized assessments fail to exhibit one or more of these characteristics that would make the test a consistently valid measure of student learning and teacher performance.  Some tests are better than others, of course, and do provide per-student/per-curricular aim reports, and others are carefully analyzed for item bias based on socio-economic or aptitude variables.  But no standardized assessment I know of is based on a prioritized curriculum or crystal-clear descriptions of learning targets.

This doesn't mean that standardized assessments have no value.  But it does mean that in terms of measuring what students know, and perhaps as importantly, the impact teachers had on what students know, their value is limited.  Well-respected tests like the National Assessment of Educational Progress (NAEP) and the Programme for International Student Assessment (PISA) are considered extremely useful for purposes of comparing overall trends in student learning across time periods and across countries.

Such comparisons of student achievement in time and place help create urgency for making sure American students have the skills they need to compete and thrive in the emerging global marketplace.  But ultimately, as Tony Wagner argues in The Global Achievement Gap, standardized tests can't assess the most important student skills, like critical thinking and creativity, necessary in today's world.  Others, responding to the Scholastic/Gates survey of teachers, agree:

Teacher-effectiveness authority Charlotte Danielson added that "not a single one of the 21st-century skills can be assessed on a multiple-choice test." She said that the appeal of standardized test scores is that they "give you a number" but that teaching is too complex to be captured in that way.

Instead, Popham and others argue for vastly improving the quality of classroom-level formative and summative assessments, which should be based on power standards and carefully-crafted and aligned learning targets, collaboratively developed whenever possible.   This kind of assessment approach is vastly more instructinally sensitive and serves as a much better gauge of student learning and teacher effectiveness.

Happily, the Primary Sources survey found teachers are advocates for this approach:

Overall, according to the report, teachers see ongoing formative assessments, class participation, and performance on class assignments as much more important measures of student learning. At the same time, most teachers (85 percent) agree that their students' growth over the course of the year should contribute significantly to evaluations of their own performance.

It's encouraging to see majorities of teachers embracing quality assessment strategies and taking ownership of the learning process.  Perhaps by focusing more heavily on these dimensions of teaching and learning and less on test preparation, performance, and analysis, educators can finally make more substantial improvements in student achievement.