Virtuous cycles and complexity in the workplace

Conventional top-down project planning and decision making approaches, in combination with systems and structures that enforce conventional hierarchical relationships work pretty well in the absence of volatility, uncertainty, and change. But the same structures that enforce order and help mitigate risk under relatively stable conditions also reduce adaptivity, which means that in our current highly complex and volatile marketplace, many conventionally structured organizations are struggling to adapt.

Several specific needs have been identified, including:

  1. Employees who embrace change and lifelong learning (especially with respect to their capacity to work with increasing complexity),
  2. Organizational cultures characterized by continuous learning & development, innovation, engagement, and collaboration within and across teams,
  3. Decision-making processes, planning processes, people development processes, and governance structures that actively support 1 & 2.

Most change processes address 1 and 2, but there has been less attention to 3. Until recently.

VCoLMany of the change processes that address number 3 involve the creative use of intentional virtuous cycles—like the one that’s at the core of our learning model (VCoL+7). Virtuous cycles like VCoL, scrum, dynamic steering, and design thinking are now being implemented in large organizations to increase agility, innovation, collaboration, learning, and engagement. And when it comes to managing complexity, they may well be the most effective tools available.

Map of Google's organizational structureAs an example, Google, which works with agile & scrum as well as other virtuous cycles, is well known for it’s culture of collaboration, continuous learning, and innovation. And its organizational structure, which eliminates silos and is sustained by cross-team collaboration, is part of what keeps that culture alive.

VCoL, like other virtuous cycles, can be embedded in organizational systems to help foster a learning culture. The classic, The fifth discipline: The art & practice of the learning organization (Peter Senge), and the more accessible, An everyone culture: Becoming a deliberately developmental organization (Robert Kegan, Lisa Laskow Lahey) describe two approaches that involve VCoLs. Lectical Assessments are designed to support approaches like these—improving performance by fostering optimal learning and development, and supporting dynamic steering by measuring program effectiveness.

Bottom of the class syndrome

What happens if you consistently punish someone for engaging in a particular practice? Most of us assume that it will get them to stop. Right? So, why have we developed an educational system that punishes millions of children for learning?

Because all capabilities are distributed in the population as “bell curves,” half of all children inevitably will be in the bottom half of their class. Most of these students will consistently receive grades that reflect poor performance relative to other students, primarily because (for a variety of reasons) they learn more slowly than their age mates. Young students tend to understand poor grades as punishments for poor performance or evidence of stupidity. The occasional low grade that is clearly attached to a lack of effort can act as an incentive to try harder. But consistently low grades with no hope of improvement teach students that learning is bad, because no matter how hard they try, it leads to punishment. I call this “bottom of the class syndrome.”

Clearly “learning is bad” is not the message teachers mean to send. They expect the punishment of a low grade to motivate students and their parents to try harder, and this is likely to work for many of the students in the upper half of the class. But it is not likely to work for kids in the bottom half of the class, because most of them are consistently slower learners. Most students in the bottom half of the class need more time and must make more effort to learn concepts and skills than students in the upper half of the class. They can’t afford to dislike learning; even more than students in the top of the class, they must retain their inborn love of learning to have any hope of success over the long term.

As long as we award scores that rank students, about half of those students will be vulnerable to “bottom of the class syndrome.” Society will lose many of them as learners, and their life choices and contributions will be unnecessarily restricted.

My colleagues and I have been working on this problem for many years. The solution we offer requires sophisticated ways of thinking about learning and motivation as well as new tools for evaluating learning—tools that don’t compare students, but allow each of them to develop on their own timeline. And it requires that we systematically study how students learn every single skill or concept we teach, so we know what every score on every assessment really means—in terms of what a given student understands and what he or she is likely to benefit from learning next. It’s hard work, but we’ve learned how to do it. Humans have tackled much more challenging problems. All we need is the will to ensure that every child has an opportunity to realize his or her potential, the patience to do the work, good people to carry the work forward, and a little thing called funding.

Virtuous cycles of learning and instruction

What is a virtuous cycle of learning?

Ideal learning occurs in virtuous cycles—repeating cycles of goal settingobservation (taking in new knowledge), testing (applying what has been learned and getting feedback on results), and reflection (figuring out which adjustments are needed to improve one’s performance on the next attempt). This process, which occurs unconsciously from birth, can be made conscious. One recent application of the virtuous cycle is in dynamic steering, in which decisions are developed, applied, and evaluated through intentionally iterating cycles. The idea is to stretch as far as possible within a given cycle, without setting immediate goals that are completely beyond one’s reach. Success emerges from the achievement of a series of incremental goals, each of which brings one closer to the final goal. Processes of this kind lay down foundational skills that support resilience and agility. For example, the infant who learns to walk also learns to fall more gracefully, which makes learning to run much less traumatic than it might have been. And decision makers who use dynamic steering learn a great deal about what makes decisions more likely to be successful, which leads to better, faster decisions in increasingly complex or thorny situations.

virtuous_cycle_with_icons_4_names

The figure on the right illustrates how educators can support virtuous learning cycles. There are 4 "steps" in this process (not necessarily in the following order):

  1. Find out what individual learners already know and how they work with their knowledge, then set provisional learning goals.
  2. Acquire and evaluate information
  3. Apply new knowledge or skills in hypothetical or real-life situations.
  4. Provide frequent opportunities for learners to reflect upon outcomes associated with the application of new knowledge in an environment in which ongoing learning, application, and reflection are consistently rewarded.

Testing as part of learning 1

Learning isn’t easy

Yet all healthy babies pursue it with dogged determination, spending hour after hour exploring—and learning to master—their own bodies, as well as their physical and social environments.

Natural testing

When infants and young children engage their environments, they receive constant feedback about what does and does not work. For example, babies spend months learning how to control the movements of their hands. An infant will spend several weeks just learning how to bring an object to her mouth. She’ll use what she learns from successes and failures to do better next time. Feedback is instant and accurate, and the results of each attempt tell her what to try next.

Babies often act like they are addicted to learning. They will tolerate an amazing amount of failure. But without prompt feedback from their external environment, they wouldn’t get far. The same is true for older children.

Testing in schools

Ideally, educational tests model natural testing by providing students with timely and accurate feedback that tells them (and their teachers) what to try next.

Learning is boring

By the third grade, when students are asked to complete the sentence, “Learning is…”, their most common answer is boring.* Many adults would agree.

Does this mean that people fundamentally dislike learning? There are those who think so, but the evidence suggests otherwise. Babies, for example, absolutely love learning. You’d think it was their reason for being. They spend just about every waking moment engaged in activities that build knowledge and skills, and are so eager to learn that they press on despite repeated, often painful failures.

Of late, my granddaughter has reminded me of this inborn passion for learning. I’ve been fascinated by how hard she has practiced to learn to manipulate objects with her hands. At the beginning, she achieved almost no success. When a toy moved into range, she waved her arms wildly. Occasionally she would make contact, but these events were so random it was hard to see how she could learn from them. Apparently, however, she learned a little from each contact, because her rate of success improved almost daily. A few months later, she can pass a toy from one hand to another, taste it, hold it out to get a better look, and bang it on the table. All her parents had to do was provide an assortment of interesting objects. This was very easy, because Erwin, as is typical of babies, finds all objects interesting.

What happens between infancy and third grade? I think we have created an educational system (and a culture) that systematically undermines many of our our children’s inborn enthusiasm for the hard work and natural pleasures of learning.

I will elaborate in future posts, starting with some comments on the effects of America’s current obsession with standardized testing and something I call “bottom of the class” syndrome.

*If you teach, check it out. Write “Learning is…” on the blackboard and ask students to complete the sentence with one word (anonymously) on a sheet of paper. Collect and tabulate their answers. Have a discussion.

Testing the limits of testing

The NTS is an interactive online survey that asks about (1) the legitimate purposes of testing and (2) how well today’s tests serve these purposes. In addition to completing a set of survey questions, respondents are offered an opportunity to write about their personal experiences with testing and share alternative testing resources. When respondents have completed the survey, they can view their results and compare them to national averages. Anyone who visits the site can read respondents’ stories, explore the resources, and track national results. Please participate in the NTS, and use your email lists and social networks to spread the word! Feel free to circulate the NTS poster or the poster announcing the NTS launch event. Contact Zachary Stein if you have questions or would like to become involved.

NTS launch event: Testing the limits of testing

Thursday, May 28th, 2009, 4:00 – 5:30 pm

Zachary Stein, Marc Schwartz, and Theo L. Dawson

The launch event will occur just prior to the opening of the second annual conference of the International Mind, Brain, and Education Society (IMBES) at the Sheraton Society Hill Hotel in Philadelphia, Pennsylvania. At this event, speakers will present preliminary data from the NTS, examine the limits of current test development methods, and explore new approaches to assessment, incorporating the perspectives of stakeholder groups who have participated in the survey so far.

More information is available on the NTS site.

Admission to the launch is FREE and open to the public, but space is limited. To attend, you must obtain a ticket from the NTS web site.

The conference will also feature a workshop on testing:

Educational testing for the 21st century: Challenges, models, and solutions

10:45 – 3:45, Saturday, May 30

Kurt Fischer, Marc Schwartz, Theo Dawson, Zachary Stein

The most basic form of educational testing takes the form of a “conversation” between an individual student and a teacher in which the student reveals what he or she is most likely to benefit from learning next. This kind of conversation increasingly takes a back seat to standardized forms of assessment that are designed to rank students for purposes that are dissociated from learning itself. Testing has lost its roots. The statistically generated rankings of standardized tests tell us very little about the specific learning needs of individual students. And it is becoming increasingly apparent that the kind of knowledge required to succeed on a typical standardized test bears little resemblance to the kind of knowledge required for adult life. The challenge we now face is creating the kind of mass-customization that revives the educative role of assessments in the local dialogue between teachers, students, and the curriculum, while maintaining the advantages of standardization. Simply stated: we need tests that help teachers meet the learning needs of individual students–tests teachers ought to teach to. In this workshop, we explore perspectives on these issues from the classroom, cognitive developmental science, psychometrics, and philosophy and offer a concrete vision for the future of assessment. The workshop is intended for educators, administrators, researchers, and policy makers. It is FREE to those who register for the entire IMBES conference. If you are interested in attending only the workshop, the fee is $80 before April 28th, and $95 after April 28th.

You can register for the conference or the workshop at the IMBES site.

Making standardized tests

There are three main players in the creation of most standardized tests. They are the (1) discipline experts, (2) item developers, and (3) psychometricians. The discipline experts are usually PhD’s who specialize in particular areas—like science, math, writing, or history. They know a lot about their content areas and have done research on teaching and learning in these areas. They also may be teachers of teachers.

A group of discipline experts work together to decide what material should be covered in lessons and on tests.* Discipline experts set standards through organizations like the National Research Council, and may or may not be affiliated with test developers. 

The item developers create test questions. They usually have a bachelor or masters degree in a particular subject area. Many have not taught, and few are experts in learning and development. Item developers design test questions that cover the content of the standards. Almost all of the items designed by item developers are multiple choice, which means that they have right and wrong answers, and thus, must focus on “factual” knowledge.

The third players are psychometricians. They put together groups of items and examine how well these work together to measure students’ knowledge of the subject at hand. Generally, psychometricians know relatively little about learning and development and do not work closely with item developers or discipline experts.

Although discipline experts may include skills for thinking and learning in their standards, these skills are not measured on standardized tests, because they cannot be evaluated with multiple choice items. And although discipline experts may focus on student understanding in their standards, research has shown that up to 50% of the students who get a multiple choice item correct cannot demonstrate understanding by providing an adequate explanation of their answer. 

Cognitive psychologists know about the problems that stem from how standardized tests are made. Many of them are not fans of standardized tests, partly because of the limitations of multiple choice items, partly because the tests are inadequately grounded in evidence about how students actually learn concepts and skills, and partly because the tests push teachers to emphasize breadth over depth and memorization of facts over skills for thinking and learning.

In future posts, after explaining a bit more about how tests work, I will examine an alternative testing model based on research into how students actually learn concepts and skills over time.

*These decisions, for the most part, are based on informed opinion and are usually not informed by systematic research into how students actually learn concepts over the course of K-12. (This is because research into how students learn concepts over time is relatively rare. Most educational research focuses on learning in a narrow age-range, attempting to identify “general mechanisms” for learning, or figuring out better ways to teach.)