Archive for category educational testing
The limitations of testing
Posted by Theo in Lectical Assessment System, educational testing, measurement, testing in general on March 15, 2010
It is important for those of us who use assessments to ensure that they (1) measure what we say they measure, (2) measure it reliably enough to justify claimed distinctions between and within persons, and (3) are used responsibly. It is relatively easy for testing experts to create assessments that are adequately reliable (2) for individual assessment, and although it is more difficult to show that these tests measure the construct of interest (1), there are reasonable methods for showing that an assessment meets this standard. However, it is more difficult to ensure that assessments are used responsibly (3).
Few consumers of tests are aware of their inherent limitations. Even the best tests, those that are highly reliable and measure what they are supposed to measure, provide only a limited amount of information. This is true of all measures. The more we hone in on a measureable dimension—in other words, the greater our precision becomes—the narrower the construct becomes. Time, weight, height, and distance are all extremely narrow constructs. This means that they provide a very specific piece of information extremely well. When we use a ruler, we can have great confidence in the measurement we make, down to very small lengths (depending on the ruler, of course). No one doubts the great advantages of this kind of precision. But we can’t learn anything else about the measured object. Its length usually cannot tell us what the object is, how it is shaped, its color, its use, its weight, how it feels, how attractive it is, or how useful it is. We only know how long it is. To provide an accurate account of the thing that was measured, we need to know many more things about it, and we need to construct a narrative that brings these things together in a meaningful way.
A really good psychological measure is similar. The LAS (Lectical Assessment System), for example, is designed to go to the heart of development, stripping away everything that does not contribute to the pure developmental “height” of a given performance. Without knowledge of many other things—such as the ways of thinking that are generally associated with this “height” in a particular domain, the specific ideas that are associated with this particular performance, information from other performances on other measures, qualitative observations, and good clinical judgment—we cannot construct a terribly useful narrative.
And this brings me to my final point: A formal measure, no matter how great it is, should always be employed by a knowledgeable mentor, clinician, teacher, consultant, or coach as a single item of information about a given client that may or may not provide useful insights into relevant needs or capabilities. Consider this relatively simple example: a given 2-year-old may be tall for his age, but if he is somewhat under weight for his age, the latter measure may seem more important. However, if he has a broken arm, neither measure may loom large—at least until the bone is set. Once the arm is safely in a cast, all three pieces of information—weight, height, and broken arm—may contribute to a clinical diagnosis that would have been difficult to make without any one of them.
It is my hope that the educational community will choose to adopt high standards for measurement, then put measurement in its place—alongside good clinical judgment, reflective life experience, qualitative observations, and honest feedback from trusted others.
What is a holistic assessment?
Posted by Theo in educational testing, measurement, research, testing in general on December 11, 2009
Thirty years ago, when I was a hippy midwife, the idea of holism began to slip into the counter-culture. A few years later, this much misunderstood notion was all the rage on college campuses. By the time I was in graduate school in the nineties there was a impassable division between the trendy postmodern holists and the rigidly old fashioned modernists. You may detect a slight mocking tone, and rightly so. People with good ideas on both sides made themselves look pretty silly by refusing, for example, to use any of the tools associated with the other side. One of the more tragic outcomes of this silliness was the emergence of the holistic assessment.
Simply put, the holistic assessment is a multidimensional assessment that is designed to take a more nuanced, textured, or rich approach to assessment. Great idea. Love it.
It’s the next part that’s silly. Having collected rich information on multiple dimensions, the test designers sum up a person’s performance with a single number. Why is this silly? Because the so-called holistic score becomes pretty-much meaningless. Two people with the same score can have very little in common. For example, let’s imagine that a holistic assessment examines emotional maturity, perspective taking, and leadership thinking. Two people receive a score of 10 that may be accompanied by boilerplate descriptions of what emotional maturity, perspective taking, and leadership attitudes look like at level 10. However, person one was actually weak in perspective-taking and strongest in leadership, and person two was weak in emotional maturity and strongest in perspective taking. The score of 10, it turns out, means something quite different for these two people. I would argue that it is relatively meaningless because there is no way to know, based on the single “holistic” score, how best to support the development of these distinct individuals.
Holism has its roots in system dynamics, where measurements are used to build rich models of systems. All of the measurements are unidimensional. They are never lumped together into “holistic” measures. That would be equivalent to talking about the temperaturelength of a day or the lengthweight of an object*. It’s essential to measure time, weight, and length with appropriate metrics and then to describe their interrelationships and the outcomes of these interrelationships. The language used to describe these is the language of probability, which is sensitive to differences in the measurement of different properties.
In psychological assessment, dimensionality is a challenging issue. What constitutes a single dimension is a matter for debate. For DTS, the primary consideration is how useful an assessment will be in helping people learn and grow. So, we tend to construct individual assessments, each of which represents a fairly tightly defined content space, and we use only one metric to determine the level of a performance. The meaning of a given score is both universal (it is an order of hierarchical complexity and phase on the skill scale) and contextual (it is provided to a performance in a particular domain in a particular context, and is associated with particular content.) We independently analyze the content of the performance to determine its strengths and weaknesses—relative to its level and the known range of content associated with that level—and provide feedback about these strengths and weaknesses as well as targeted learning suggestions. We use the level score to help us tell a useful story about a particular performance, without claiming to measure “lenghtweight”. This is accomplished by the rigorous separation of structure (level) and content.
*If we described objects in terms of their lengthweight, an object that was 10 inches long and 2 lbs could have a lengthweight of 12, but so could an object that was 2 inches long and 10 lbs.
Teacher pay and standardized test results
Posted by Theo in educational testing, standardized testing, teaching on November 23, 2009
At the end of October, the Century Foundation released a paper entitled, Eight reasons not to tie teacher pay to standardized test results. I agree with their conclusions, and would add that even if all standardized tests were extremely reliable and measured exactly what they intended to measure, this would be a bad idea. This is because success in the adult world requires a multiplicity of skills and forms of knowledge, and tests focus on only some of these, one at a time. Until we can construct multifaceted longitudinal stories about the progress of individual students that are tied to a non-arbitrary standardized metric, we should not even consider linking student evaluations to teacher pay.
What is a developmental assessment?
Posted by Theo in cognitive development, educational testing, testing in general on July 29, 2009
A developmental assessment is a test of knowledge and thinking that is based on extensive research into how students come to learn specific concepts and skills over time. All good developmental assessments require test-takers to show their thinking by making written or oral arguments in support of their judgments. Developmental assessments are less concerned about “right” answers and more concerned with how students use their knowledge and thinking skills to solve problems. A good developmental assessment should be educative in the sense that taking it is a learning experience in its own right, and each score is accompanied by feedback that tells students what they are most likely to benefit from learning next.
Reliability 2: How high should it be?
Posted by Theo in educational testing, standardized testing, testing in general on July 5, 2009
There is a great deal of confusion in the assessment community about the interpretation of statistical reliability. This confusion results in part from the different ways in which researchers and test developers approach the issue. Researchers learn how to design research instruments which they use to study population trends or compare groups. They evaluate the quality of their instruments with statistics. One of the statistics used is Cronbach’s Alpha, an indicator of statistical reliability that ranges from 0 to 1. Researchers are taught that Alphas above .77 or so are acceptable for their instruments, because this level of reliability ensures that their instrument is measuring real differences between people.
Test developers use a special branch of statistics called psychometrics to build assessments. Assessments are designed to evaluate individuals. Like researchers, test developers are concerned about reliability, but for somewhat different reasons. From a psychometric point of view, it is not enough to know that an assessment measures real differences between people. Psychometricians need to be confident that the score awarded to an individual is a good estimate of that particular individual’s true score. Because of this, most psychometricians set higher standards for reliability than those set by researchers.
The table below will help to clarify why it is important for assessments to have higher reliabilities than research instruments. It shows the relationship between statistical reliability and the number of distinct levels (strata) a test can be said to have. For example, an assessment with a reliability of .80, has 3 strata, whereas an assessment with a reliability of .94 has 5.
| Reliability | Strata |
| .70 | 2 |
| .80 | 3 |
| .90 | 4 |
| .94 | 5 |
| .96 | 7 |
| .97 | 8 |
| .98 | 9 |
Strata have direct implications for the confidence we can have in a specific person’s score on a given assessment, because they tell us something about the range within which a person’s true score would fall, given a particular score. Imagine that you have taken a test with a scoring range of 0 to 500 and a reliability of .94. The number of strata into which this assessment can be divided is 5, which means that each strata equals about 100 points on the 500 point scale. If your score on this test is 350, your true score is likely to fall within the range of 300 to 400*.
Statistical reliability is only one of the ways in which assessments should be evaluated. Test developers should also ask how well an assessment measures what it is intended to measure. And those who use an assessment should ask whether or not what it measures is relevant or important.
*This range will be wider at the top and bottom of the scoring range and a bit narrower in the middle of the range.
References
Guilford J. P. (1965). Fundamental statistics in psychology and education. 4th Edn. New York: McGraw-Hill.
Kubiszyn T., Borich G. (1993). Educational testing and measurement. New York: Harper Collins.
Wright B. D. (1996). Reliability and separation. Rasch Measurement Transactions, 9, 472.
A good test
Posted by Theo in cognitive development, educational testing, learning, motivation, testing in general on April 29, 2009
In this post, I explore a way of thinking about testing that would lead to the design of tests that are very different from most of the tests students take today.
Two propositions, an observation, and a third proposition:
Proposition 1. Because adults who do not enjoy learning are at a severe disadvantage in a rapidly changing world, an educational system should do everything possible to nurture children’s inborn love of learning.
Proposition 2. In K-12, the specific content of a curriculum is not as important as the development of broadly applicable skills for learning, reasoning, communicating, and participating in a civil society. (The content of the curriculum would be chosen to support the development of these skills and could—perhaps should—differ from classroom to classroom.)
Observation. Testing tends to drive instruction.
Proposition 3. Consequently, tests should evaluate relevant skills and be employed in ways that support students’ natural love of learning.
Given these propositions, here is my favorite definition of a “good test.”
A good test is part of the conversation between a “student” and a “teacher” that tells the teacher what the student is most likely to benefit from learning next.
I’ll unpack this definition and show how it relates to the proposals listed above:
Anyone who has carefully observed an infant in pursuit of knowledge will understand the conversational nature of learning. A parent holds out a shiny spoon and an infant’s arms wave wildly. Her hand makes contact with the spoon and a message is sent to her brain, “Something interesting happened!” The next day, her arm movements are a little less random. She makes contact several times, feeling the same sense of satisfaction. Her parents laugh with delight. She coos. In this way, her physical and social environment provide immediate feedback each time she succeeds (or fails). Over time, the infant uses this information to learn how to reach out and touch the spoon at will. Of course, she is not satisfied with merely touching the spoon, and, through the same kind of trial and error, supplemented with a little support from Mom and Dad, she soon learns to bring the spoon to her mouth. And the conversation goes on.
Every attempt to touch the spoon is a kind of test. Every success is an affirmation that the strategy just employed was an effective strategy, but the story does not end here. In her quest to master her environment, the infant keeps moving the bar. Once she can do so at will, touching the spoon is no longer satisfying. She moves on to the next skill—holding the spoon, and the next—bringing it to her mouth, etc. Having observed this process hundreds of times, I strongly suspect that a sense of mastery is the intrinsic reward that motivates learning, while conversation, including both social and physical interactions, acts as the fuel.
Conversation
A good educational test should have the same quality of conversation, in the form of performance and feedback, that is illustrated in the example above. In an ideal testing situation, the student shows a teacher how he or she understands new concepts and skills, then the teacher uses this information to determine what comes next.
Part of the conversation
However, a good test is part of the conversation—not the entire conversation. No single test (or kind of conversation) will do. For example, the infant reaches for the spoon because she finds it interesting, and she must be interested enough to reach out many dozens of times before she can grasp an object at will. Good parents recognize that she expresses more sustained interest if they provide her with a number of different objects—and don’t try to force her to manipulate objects when she would rather be nursing or sleeping. Each act is a test embedded in a long conversation that is further embedded in a broader context.
What comes next?
In the story, I suggest that the spoon must be both interesting and within an infant’s reach before it can become part of an ongoing conversation. In the same way, a good test should both be engaging and within a student’s reach in order to play its role in the conversation between student and teacher.
An engaging test of appropriate skills can tell us how a student understands what he or she is learning, but this knowledge, by itself, does not tell the teacher (or the student) what comes next. To find out, researchers must study how particular concepts and skills are learned over time. Only when we have done a good job describing how particular skills and concepts are learned can we predict what a student is most likely to benefit from learning next.
So, a good test must not only capture the nature of a particular student’s understanding, it must also be connected to knowledge about the pathways through which students come to understand the concepts and skills of the knowledge area it targets.
Back to conversation
I argue above, that in infancy, a sense of mastery is the intrinsic reward that motivates learning, while conversation is the fuel. If conversation is the fuel, tests that do a good job serving the conversational function I outline here are likely to fuel students’ natural pursuit of mastery and a lifelong love of learning.
Later: But what about accountability?
Motivation & standardized testing
Posted by Theo in educational testing, motivation, standardized testing, teaching on April 25, 2009
Check out this post at Docere est Discere (Musings on language and teaching).
Test validity & tacit knowledge
Posted by Theo in decision making, educational testing, learning, reasoning on April 21, 2009
As you probably know if you are reading this post, my colleagues and I make developmental assessments, several of which are focused on adult skills like managerial decision making. I am often asked about the validity of these assessments as it pertains to the distinction between intuitive or tacit knowledge and the kind of skills people need to do well on a developmental assessment. The short answer is that tacit knowledge is not captured by any assessment that asks people to reason through a problem, because tacit knowledge is, well, tacit.
How does it work?
Often, people know more about a particular subject than they can communicate verbally. This is because much of what we learn through experience does not automatically become part of our conscious knowledge. It’s in a form that’s difficult to share. We call this kind of knowledge tacit or implicit. You can also think of it as intuitive. Tacit knowledge is not a bad thing. It helps us make quick choices in familiar situations; we’d be in big trouble if we had to think through every single situation in our lives before we made a decision! But tacit knowledge has its limits.
First, because we aren’t able to bring it into focus, we can’t share tacit knowledge with others. This is the case in a business where the person at the top is an “intuitive” leader. Because his or her leadership skills are tacit, they can’t be shared. In situations like this, it is not uncommon for an institution to last only as long as its leader.
Second, tacit knowledge is limited by our direct experience, which means it isn’t terribly useful for dealing with novelty or abstractions. Because tacit knowledge is experienced-based, it is most useful in situations that are like those we have confronted in the past. Unfortunately for those who rely too heavily on tacit knowledge, the modern world constantly provides us with new challenges. To meet them, we need conscious methods for evaluating knowledge and experience.
Third, sometimes our tacit knowledge is sub-optimal. For example, people who learn the skills required to survive on the rough side of town or in an abusive relationship often fare poorly when they try to function outside of those contexts, partly because their tacit knowledge isn’t useful in their new surroundings. Before they can learn more adaptive skills, they usually need to bring their tacit knowledge into consciousness.
Finally, relying on tacit knowledge limits our development. When we habitually rely on our tacit knowledge, learning not only slows down, it’s quality is compromised. Optimal learning requires that we reflect consciously upon our own experiences and actively seek other perspectives to fill in the gaps.
Implications for assessment
The implications for assessment are clear. If knowledge is tacit, it won’t be reflected in an assessment of reasoning skills. Although good developmental assessments can provide accurate evaluations of the level of complexity a person can consciously cope with in a specific skill area, they can’t tell us everything we need to know about a person’s capabilities. This is one good reason—in a much longer list of reasons—to avoid making high stakes decisions on the basis of a single form of evaluation.
Predicting trends, testing people
Posted by Theo in educational testing, standardized testing, testing in general on April 15, 2009
Mark Forman, in his response to the post entitled, IQ and development, wrote about the difference between predicting trends and testing individuals. I agree that people, including many academics, do not understand the difference between using assessments to predict trends and using assessments to make judgments about individuals. There are two main issues: First, as Mark argues, questions of validity differ, depending upon whether we are looking at individuals or population trends. If we are looking at trends, determining predictive validity is a simple matter of determining if an assessment helps an institution make more successful decisions than it was able to make without the assessment. However, if a test is intended to be useful to individuals (aid in their learning, help them determine what to learn next, help them find the best place to learn, help them decide what profession to pursue, etc.), predictive validity cannot be determined by examining trends. In this case, the predictive validity of an assessment should be evaluated in terms of how well it predicts what individual test-takers can most benefit from learning next, where they can learn it, or what kind of employment they should seek—as individuals.
The second issue concerns reliability. Especially in the adult assessment field, researchers often do not understand that the levels of statistical reliability considered acceptable for studies of population trends are far from adequate for making judgments about individuals. Many of the adult assessments that are on the market today have been developed by researchers who do not understand the reliability criteria for assessments used to test individuals*. As a consequence, the reliability of these assessments is often so low that we cannot be confident that a score on a given assessment is truly different from any other score on that assessment.
*Unfortunately, there is no magic reliability number. But here are some general guidelines. The absolute minimum statistical reliability for an assessment that claims to distinguish two or three levels of performance is an alpha of .85. To claim up to 6 levels, you need an alpha of .95. You will also want to think about the meaning of these distinctions between levels in terms of confidence intervals. A confidence interval is the range in which an individual’s true score is most likely to fall. For example, in the case of Lectical™ assessments, the statistical reliabilities we have calculated over the last 10 years indicate that the confidence interval around Lectical scores is generally around 1/4 of a level (a phase).
Advice: If statistical reliability is not reported (preferably in a peer reviewed article), don’t use the test.
IQ and development
Posted by Theo in cognitive development, educational testing, testing in general on April 14, 2009
IQ is a dimension of ability that has been defined using a form of statistical modeling called psychometrics. It is based entirely on psychometric analysis of results from tests consisting of many items, each of which has one correct answer.
IQ scores are arranged along a scale that is based upon the performances of hundreds of people who have taken the same test.
IQ is considered to be a relatively fixed characteristic of a person. People who score higher on an IQ test are considered to be more intelligent than people who score lower.
Cognitive development is a theoretically defined, evidence based dimension. Developmental level is determined by asking individuals to engage in activities that expose their reasoning. Items on developmental assessments are typically open-ended and do not focus on correct answers. They focus on how people go about seeking answers.
A single developmental dimension has been shown to underlie development in a wide range of cognitive domains, making it possible to define a non-arbitrary scale along which development progresses. Individual performances can be placed within a range on this scale.
Cognitive developmental level is not viewed as a fixed trait and is known to vary within persons, depending on knowledge area and a range of contextual variables. Individuals who demonstrate higher levels of cognitive development are viewed as more cognitively developed than those demonstrating lower levels of cognitive development.
The relation between IQ and cognitive development
Children with higher IQ’s learn the kind of knowledge and skills represented in IQ tests earlier than people with lower IQ’s. There is some evidence that cognitive development is likely to be more rapid (and have a higher “endpoint”) in people who have higher IQ’s.
Limitations of testing
The subject matter of IQ tests is limited, and the skill sets that are tested are narrow, so we have to be careful about making generalizations about people based on test results—especially the results of single tests. The same is true for cognitive developmental assessments. Good cognitive developmental assessments are now providing scores with a level of precision similar to that of conventional assessments, but even the most precise and accurate scores apply to performance on a single assessment in a single subject area, and do not capture the full range of capabilities of a test-taker.
The inability of any single assessment (or type of assessment) to provide an accurate account of the capabilities of an individual suggests that the best (most ethical) use of assessments involves repeated measurements across a wide range of subject areas over time.
Recent Comments