Task demands and capabilities (the complexity gap)

For decades, my colleagues and I have been working with and refining a developmental assessment system called the Lectical Assessment System (now also an electronic scoring system called CLAS). It can be used to score (a) the complexity level of people’s arguments and (b) the complexity level—“task demands”—of specific situations or roles. For example, we have analyzed the task demands of levels of work in large organizations and assessed the complexity level of employees’ thinking in several skill areas — including reflective judgment/critical thinking and leadership decision-making.

The figure on the left shows the relation between the task demands of 7 management levels and the complexity level scores received on an assessment of decision making skills taken by leaders occupying these positions. The task demands of most positions increase in a linear fashion, spanning levels 10–13 (a.k.a. 1000–1399).

After work level 2 (entry level management), the capabilities of leaders do not, for the most part, rise to these task demands.

This pattern is pervasive—we see it everywhere we look—and it reflects a hard truth. None of us is capable of meeting the task demands of the most complex situations in today's world. I've come to believe that in many situations our best hope for meeting these demands is to (1) recognize our human limitations, (2) work strategically on the development of our own skills and knowledge, (3) learn to work closely with others who represent a wide range of perspectives and areas of expertise, and (4) use the best tools available to scaffold our thinking.


We aren't alone. Others have observed and remarked upon this pattern:

Jaques, E. (1976). A general theory of bureaucracy. London: Heinemann Educational.

Habermas, J. (1975). Legitimation crisis (T. McCarthy, Trans.). Boston: Beacon Press.

Kegan, R. (1994). In over our heads: The mental demands of modern life. Cambridge, MA: Harvard University Press.

Bell, D. (1973) The coming of post-industrial society. New York: Basic Books

Please follow and like us:

Testing as part of learning 2

I can’t help it, I’m a developmental psychologist. I’ve been lurking about, watching my Granddaughter, Erwin, as she learns to master her environment. She’s about 8 months old now (real age, she was three months premature, so her birth age is 11 months)

Last week, Erwin figured out that complex actions can be used intentionally to make things happen in social situations. For example, she started reaching toward her Mom and Dad to indicate her intention to be picked up. At around same time, she began pointing to objects to indicate interest or draw them to the attention of her others. And she has begun to imitate actions like waving, clapping, and head shaking. Today, when we were Skyping, she clapped her hands to get me to play pat-a-cake, and she shakes her head to get her Mom to do the same—which she finds hilarious. To Mom’s dismay, Erwin is so excited by this new way of influencing her environment that she has stopped napping.

To see an example of Erwin’s attempts at verbal communication and her new reaching behavior, double-click on the picture below. Notice how emphatic her arm extension is, and how she makes eye contact as she reaches out.

A few months ago, most of Erwin’s actions were aimed toward physical mastery—learning to obtain objects and manipulate them in a variety of ways, learning to move herself toward things she wanted to manipulate, or playing with sound just to hear the results.

When she was learning to do physical things, the physical environment provided most of the feedback. Although her parents were there to give encouragement, we all had the sense that it was the physical feedback that she craved—getting an object to her mouth, inching toward a favorite toy, pulling herself to stand.

Now she craves feedback from her parents; she has shifted her focus from physical mastery to social mastery. She reaches for Mom and gets picked up. She shakes her head and Mom shakes her head back. She points to a banana, and Dad brings it to her. She claps her hands, and Grandma plays pat-a-cake. And every time she undertakes a new action, she is conducting a test.

Testing is part of learning.

Each time any infant tries out a new skill, she is conducting a test. Each attempt is part of an action-feedback loop. Repeated attempts to master a new skill form a series of these action-feedback loops. Each iteration is an exemplary test—in the sense that it is educative—that guides the infant incrementally toward a new level of mastery.

Interestingly, infants never tire of this kind of testing, even when the feedback is not instantly gratifying. In fact, much of the feedback is along the lines of, “almost, but not quite,” or “that didn’t work,” neither of which seem to get in the way of infant learning. For example, when Erwin first started reaching toward her parents to ask to be picked up, her action was not easy to read. It rarely got the desired response. She gradually learned that the reaching needed to be clearly directed toward the parent and accompanied by eye contact. Now the message is, “You’ve got it!” At this point, Erwin takes the skill for granted, and has shifted her attention to things she has not yet mastered, like figuring out how to get adults to do other interesting or gratifying things.

The natural action-feedback mechanism of infancy works perfectly, because the proverbial carrot is usually, due to the very nature of normal human environments, dangled at just the right distance. Good parents respond to early attempts at communication, rewarding them with interesting responses, but success isn’t the only reward; it’s always accompanied by a new “carrot”—another interesting possibility just beyond the infant’s reach. In this way, the action-feedback mechanism functions both as an aid to learning and as a motivator.

Aspects of this “carrot-and-stick” perspective on learning have been expanded and described in a variety of research traditions—e.g., as part of the notion of reinforcement feedback in social learning theory (Bandura, 1977), as zone of proximal development in Vygotsky’s (1986) work, and as part of a complex process of assimilation and accommodation in Piaget’s (1985) work. It is important, because it speaks both to how we learn and to our motivation for learning. Good feedback plays two essential roles. First, it helps the learner decide what to try next. Second, it motivates the learner to keep striving toward mastery. And, as the infant example suggests, feedback cannot be reduced to simple reward or punishment. Ideally, it is information that supports learning by being useful to the learner. Learners are not motivated by reward or punishment per se, but by an optimal combination of “not there yet” “almost” and “you’ve got it”.

DiscoTests are for learning

Most of today’s tests provide feedback in the form of rewards (good grades, advancement, or honors) or punishment (bad grades and failure). My colleagues and I don’t find this acceptable, so we’ve created a nonprofit called DiscoTest. The overarching objective of the DiscoTest Initiative is to contribute to the development of optimal learning environments by creating assessments that deliver the kind of educative feedback that learners need to learn optimally. DiscoTests determine where students are in their individual learning trajectories and provide feedback that points toward the next incremental step toward mastery.

I’ll be writing more about DiscoTest in future posts. For now, if you’d like to know more, please visit the DiscoTest web site.

Please follow and like us:

Kegan’s Subject-Object Interview and the LSUA

Before I write about the relation between Kegan's Subject-Object Interview and the LSUA (the Lectical Self-Understanding Assessment), I'd like to explain some differences between these assessments. First, the SOI is both an interview and an assessment system. It was developed by studying the interviews of a small sample of respondents (Does anyone know how many?) who were interviewed on several occasions over the course of several years (Again, does anyone know how many or how often?). The level definitions and the scoring criteria in the SOI are tied to the subject matter of the interviews in the original sample (construction sample). For this reason, the SOI is called a domain-specific assessment. Researchers would say that the levels were defined by "bootstrapping" from the longitudinal data. Critiques of this kind of assessment point to bias in their level definitions (due to their small and culturally narrow construction samples), the related conflation (confusion) of particular conceptual content with developmental levels, and a weak articulation of the lowest levels, which are not based on direct empirical evidence from appropriate-aged respondents.

With respect to the LSUA, I want to clarify that it is scored with the Lectical Assessment System (LAS), a content-independent developmental scoring system that was created, in part, by identifying the dimension that underlies all longitudinally bootstrapped developmental assessment systems*. The SOI was one of the assessment systems I studied on the way to developing the LAS. Consequently, if the LAS does what it is supposed to do, it should capture the developmental dimension that underlies Kegan's system even better than his scoring system, because the LAS is a second generation developmental scoring system that is not restrained by a content-driven scoring process (Dawson, 2002; Dawson, Xie, & Wilson, 2003: There is much written about this in our published work, available on our web site.)

What is the relation between the LSUA and the Subject-Object Interview?

This is a difficult question to answer, partly because there is no research that directly compares the SOI and the LSUA. However, because the LAS is a domain independent scoring system that can be used to score any text that includes judgments and justifications, I have used it to score the SOI scoring manual. The developmental sequence for SOI levels 3 to 5 corresponds well to the dimension captured by the LAS, and levels 3-5 correspond roughly with Lectical Levels 10-12. However, Kegan's lower levels do not match up as well, possibly because his construction sample (the sample used to define his levels), as far as we can determine, did not include young children. (Kegan's original research was never published in a form that would allow us to evaluate the approach he took to defining his levels or the reliability and validity of the SOI. All we can locate are a few very small studies of inter-rater reliability, most of which are unpublished [Kegan, 2002].)

Comparisons of the Subject-Object Interview with other developmental assessment systems

There is some research comparing the SOI with other developmental assessment systems. In general, this research finds that the SOI and these other systems are likely to tap the same developmental dimension (see Pratt, et. al., 1991).

Ideally, we would like to conduct a direct comparison of the LAS and the scoring system Kegan developed to score the SOI, as we have done with other developmental assessment systems. (We are working with a graduate student who is planning do do this kind of comparison.) In the mean time, we can point to comparisons between the LAS and several other developmental assessment systems (Kohlberg, Armon, Kitchener & King, Perry) that were developed using methods similar to those used by Kegan, and have routinely found strong correlations (above .85) between these scoring systems and the LAS, especially when they are used to score the same material (Dawson, 2000, 2001 2002a, 2004; Dawson, Xie, & Wilson, 2003 ).

Finally, some of Kegan's level definitions are almost identical to those of Kohlberg and Selman. In fact, I would argue that they are primarily an extension of Selman's original work on socio-moral perspective, which has informed most domain-based developmental assessment systems (including all of the systems mentioned here) since it was introduced in the 1960's (and was a great help to me when I was developing the LAS).

*The claim that there is a single developmental dimension that underlies these systems is NOT the same thing as a claim that an individual will be at the same level in different knowledge/skill areas.

References

Commons, M. L., Armon, C., Richards, F. A., Schrader, D. E., Farrell, E. W., Tappan, M. B., et al. (1989). A multidomain study of adult development. In D. Sinnott, F. A. Richards & C. Armon (Eds.), Adult development, Vol. 1: Comparisons and applications of developmental models. (pp. 33-56). New York: Praeger Publishers.

Dawson, T. L. (2000). Moral reasoning and evaluative reasoning about the good life. Journal of Applied Measurement, 1(4), 372-397.

Dawson, T. L. (2001). Layers of structure: A comparison of two approaches to developmental assessment. Genetic Epistemologist, 29, 1-10.

Dawson, T. L. (2002a). A comparison of three developmental stage scoring systems. Journal of Applied Measurement, 3, 146-189.

Dawson, T. L. (2002b). New tools, new insights: Kohlberg’s moral reasoning stages revisited. International Journal of Behavioral Development, 26, 154-166.

Dawson, T. L., Xie, Y., & Wilson, M. (2003). Domain-general and domain-specific developmental assessments: Do they measure the same thing? Cognitive Development, 18, 61-78.

Dawson, T. L. (2004). Assessing intellectual development: Three approaches, one sequence. Journal of Adult Development, 11, 71-85.

Kegan, R. (2002). A guide to the subject-object interview. Unpublished Scoring manual. Harvard Graduate School of Education.

King, P. M., Kitchener, K. S., Wood, P. K., & Davison, M. L. (1989). Relationships across developmental domains: A longitudinal study of intellectual, moral, and ego development. In M. L. Commons, J. D. Sinnot, F. A. Richards & C. Armon (Eds.), Adult development. Volume 1: Comparisons and applications of developmental models (pp. 57-71). New York: Praeger.

Lambert, H. V. (1972). A comparison of Jane Loevinger's theory of ego development and Lawrence Kohlberg's theory of moral development. University of Chicago, Chicago, IL.

Pratt, M. W., Diessner, R., Hunsberger, B., Pancer, S. M., & Savoy, K. (1991). Four pathways in the analysis of adult development and aging: Comparing analyses of reasoning about personal-life dilemmas. Psychology & Aging, 6, 666-675.

Sullivan, E. V., McCullough, G., & Stager, M. A. (1970). A developmental study of hte relationship between conceptual, ego, and moral development. Child Development, 41, 399-411.

Please follow and like us: