This morning, I received a newsletter from Sir Ken Robinson, a popular motivational speaker who focuses on education. There was a return email address, so I wrote to him. Here's what I wrote:
Dear Sir Ken,
"I love your message. I'm one of the worker bees who's trying to leverage the kind of changes you envision.
After 20+ years of hard work, my colleagues and I have reinvented educational assessment. No multiple choice. No high stakes. Our focus is on assessment for learning—supporting students in learning joyfully and deeply in a way that facilitates skills for learning, thinking, inquiring, relating and otherwise navigating a complex world. Our assessments are scalable and standardized, but they do not homogenize. They are grounded in a deep study of the many pathways through which students learn key skills and concepts. We're documenting, in exquisite (some would say insane) detail, how concepts and skills develop over time so we can gain insight into learners' knowledge networks. We don't ask about correctness. We ask about understanding and competence and how they develop over time. And we help teachers meet students "where they're at."
We've accumulated a strong base of evidence to support these claims. But now that we're ready to scale, we're running up against hostility toward all standardized assessment. It's difficult to get to the point where we can even have a conversation with our pedagogical allies. Ouch!
Lectica is organized as a nonprofit so we can guarantee that the underprivileged are served first. We plan to offer subscriptions to our assessments (learning tools) without charge to individual teachers everywhere.
We've kept our heads down as we've developed our methods and technology. Now we're scaling and want to be seen. We know we're part of the solution to today's educational crisis—perhaps a very big part of the solution. I'm hoping you'd like to learn more."
My email was returned with this message: "The email account that you tried to reach does not exist." How frustrating.
So, I thought I'd pen this post and ask my friends and colleagues to help me get access to Sir Ken's ear. If you know him, please forward this message. I'm certain he'll be interested in what we're doing for learning and development. Where are you Sir Ken Robinson? Can you hear me? Are you out there?
For several years now, one of our heroes, professor Howard Drossman of Colorado College and the Catamount Center, has been working with Lectical Assessments and helping us build LESA, the Lectical Environmental Stewardship Assessment.
Dr. Drossman's areas of expertise include developmental pedagogy, environmental stewardship, and the development of reflective judgment. His teaching focuses on building knowledge, skill, and passion through deep study, hands-on experience, and reflection.
For example, Dr. Drossman and ACM (Associated Colleges of the Midwest) offered a 10-day faculty seminar on interdisciplinary learning called Contested Spaces. This physically and intellectually challenging expeditionary learning experience provided participants with multiple disciplinary perspectives on current issues of land stewardship in the Pikes Peak region of Colorado.
A second, ongoing program is offered by Catamount Center and Colorado College is dedicated to inspiring the "next generation of ecological stewards." This program, called TREE (Teaching & Research in Environmental Education), is a 16-week, residential program for undergraduate students who have an interest in teaching and the environment. Program participants live and learn in community at the Catamount Mountain Campus, which is locatedin a montane forest outside of Woodland Park, Colorado. Through study and practice, they cultivate their own conceptions of environmental stewardship and respect for the natural world, while building skills for creating virtuous cycles of learning and useable knowledge in K-12 classrooms.
Dr. Drossman embeds Lectical Assessments in both of these programs, using them to customize instruction, support individual development, and measure program outcomes. He also is working closely with us on the development of the LESA, which is one of the first assessments we plan to bring online after our new platform, LecticaLive, has been completed.
In a recent blog post—actually in several recent blog posts—I've been emphasizing the importance of building tomorrow's skills. These are the kinds of skills we all need to navigate our increasingly complex and changing world. While I may not agree that all of the top 10 skills listed in the World Economic Forum report (shown above) belong in a list of skills (Creativity is much more than a skill, and service orientation is more of a disposition than a skill.) the flavor of this list is generally in sync with the kinds of skills, dispositions, and behaviors required in a complex and rapidly changing world.
The "skills" in this list cannot be…
developed in learning environments focused primarily on correctness or in workplace environments that don't allow for mistakes; or
These "skills" are best developed through cycles of goal setting, information gathering, application, and reflection—what we call virtuous cycles of learning—or VCoLs. And they're best assessed with tests that focus on applications of skill in real-world contexts, like Lectical Assessments, which are based on a rich research tradition focused on the development of understanding and skill.
“Why measure growth in complexity level?”* That’s the question a new acquaintance asked me recently, and it took me by surprise.
One lesson I’ve had to learn again and again is that the only way to escape the boundaries of my own perspective is to listen hard to the perspectives of others. These perspectives are often reflected in the questions people ask.
“Why measure growth?” I didn’t realize this question needed an answer. Like other questions that have surprised me, this one relates to something I take for granted — one of the fundamental assumptions that underlie my research.
Why we do it
We measure growth for three primary reasons — to make it visible, to learn how people learn, and to customize learning.
By measuring growth, we make it visible. Being able to see evidence of our own growth motivates us to grow further. Contrast this with receiving grades or conventional test scores. They compare you with other people. If you get good grades, you may be motivated to strive for even better grades. But if you consistently get poor grades, you’re more likely to feel like you’re being punished for your learning efforts. By measuring growth, we provide positive motivation for every learner.
A measure of growth helps us understand how people learn. At Lectica, we’re constantly asking how particular growth scores relate to specific skills and knowledge, and how current skills and knowledge relate to the skills and knowledge we observe at the next level. In other words, we’re systematically and continually documenting the development of knowledge and skills so we can answer questions like, “Is there an optimal way for people to learn this skill?”
A measure of growth allows us to determine what a learner is most likely to benefit from learning next. This is important, because when we get the difficulty of the next learning challenge just right (not to easy and not too hard), we activate the brain’s inborn motivational system. This not only increases motivation in the moment, but supports a lifelong love of learning while increasing the rate of development.
These aren’t the only reasons for measuring growth in complexity level, but they are the reasons at the core of our work. Measures of complexity level can also be used to help match people to roles, as in recruitment, or deciding which political candidate is best equipped to handle the complexity of a particular office.
*When we talk about measuring growth, we don't mean the ability to get more items right on a multiple choice test. We measure developmental growth—growth in the level skill with which people apply their knowledge in complex real-world contexts.
I’ve been auditing a very popular 4.5 star Coursera course called “Learning how to learn.” It uses all of the latest research to help people improve their “learning skills.” Yet, even though the lectures in the course are interesting and the research behind the course appears to be sound, I find it difficult to agree that it is a course that helps people learn how to learn.
First, the tests used to determine how well participants have built the learning skills described in this course are actually tests of how well they have learned vocabulary and definitions. As far as I can tell, no skills are involved other than the ability to recall course content. This is problematic. The assumption that learning vocabulary and definitions builds skill is unwarranted. I believe we all know this. Who has not had the experience of learning something well enough to pass a test only to forget most of what they had learned shortly thereafter?
Second, the content in tests at the end of the videos aren’t particularly relevant to the stated intention of the course. These tests require remembering (or scrolling back to) facts like “Many new synapses are formed on dendrites.” We do not need to learn this to become effective learners. The test item for which this is the correct answer is focused on an aspect of how learning works rather than how to learn. And although understanding how learning works might be a step toward learning how to learn, answering this question correctly doesn’t tell us how the participant understands anything at all.
Third, if the course developers had used tests of skill—tests that asked participants to show off how effectively they could apply described techniques, we would be able to ask about the extent to which the course helps participants learn how to learn. Instead, the only way we have to evaluate the effectiveness of the course is through participant ratings and comments—how much people like it. I’m not suggesting that liking a course is unimportant, but it’s not a good way to evaluate its effectiveness.
Fourth, the course seems to be primarily concerned with fostering a kind of learning that helps people do better on tests of correctness. The underlying and unstated assumption seems to be that if you can do better on these tests, you have learned better. This assumption flies in the face of several decades of educational research, including our own [for example, 1, 2, 3]. Correctness is not adequate evidence of understanding or real-world skill. If we want to know how well people understand new knowledge, we must observe how they apply this knowledge in real-world contexts. If we want to evaluate their level of skill, we must observe how well they apply the skill in real-world contexts. In other words, a course in learning how to learn—especially a course in learning how to learn—should be building useable skills that have value beyond the act of passing a test of correctness.
Fifth, the research behind this course can help us understand how learning works. At Lectica, we’ve used the very same information as part of the basis for our learning model, VCoL+7. But instead of using this knowledge to support the status quo—an educational system that privileges correctness over understanding and skill—we’re using it to build learning tools designed to ensure that learning in school goes beyond correctness to build deep understanding and robust skill.
For the vast majority of people, schooling is not an end in itself. It is preparation for life—preparation with tomorrow’s skills. It’s time we held our educational institutions accountable for ensuring that students know how to learn more than correct answers. Wherever their lives take them, they will do better if equipped with understanding and skill. Correctness is not enough.
 FairTest; Mulholland, Quinn (2015). The case against standardized testing. Harvard Political Review, May 14.
 Schwartz, M. S., Sadler, P. M., Sonnert, G. & Tai, R. H. (2009). Depth versus breadth: How content coverage in high school science courses relates to later success in college science coursework. Science Education, 93, 5, 798-826.
During the 70s and 80s I practiced midwifery. It was a great honor to be present at the births of over 500 babies, and in many cases, follow them into childhood. Every single one of those babies was a joyful, driven, and effective "every moment" learner. Regardless of difficulty and pain they all learned to walk, talk, interact with others, and manipulate many aspects of their environment. They needed few external rewards to build these skills—the excitement and suspense of striving seemed to be reward enough. I felt like I was observing the "life force" in action.
Unfortunately as many of these children approached the third grade (age 8), I noticed something else—something deeply troubling. Many of the same children seemed to have lost much of this intrinsic drive to learn. For them, learning had become a chore motivated primarily by extrinsic rewards and punishments. Because this was happening primarily to children attending conventional schools (Children receiving alternative instruction seemed to be exempt.) it appeared that something about schooling was depriving many children of the fundamental human drive required to support a lifetime of learning and development—a drive that looked to me like a key source of happiness and fulfillment.
Understanding the problem
Following upon my midwifery career, I flirted briefly with a career in advertising, but by the early 90's I was back in school—in a Ph.D. program in U. C. Berkeley's Graduate School of Education—where I found myself observing the same pattern I'd observed as a midwife. Both the research and my own lab experience exposed the early loss of students' natural love of learning. My concern was only increased by the newly emerging trend toward high stakes multiple choice testing, which my colleagues and I saw as a further threat to children's natural drive to learn.
Most of the people I've spoken to about this problem have agreed that it's a shame, but few have seen it as a problem that can be solved, and many have seen it as an inevitable consequence of either mass schooling or simple maturation. But I knew it was not inevitable. Children and those educated in a range of alternative environments did not appear to lose their drive to learn. Additionally, above average students in conventional schools appeared to be more likely to retain their love of learning.
I set out to find out why—and ended up on a long journey toward a solution.
How learning works
First, I needed to understand how learning works. At Berkeley, I studied a wide variety of learning theories in several disciplines, including developmental theories, behavioral theories, and brain-based theories. I collected a large database of longitudinal interviews and submitted them to in-depth analysis, looked closely at the relation between testing and learning, and studied psychological measurement, all in the interest of finding a way to support childrens' growth while reinforcing their love of learning.
My dissertation—which won awards from both U.C. Berkeley and the American Psychological Association—focused on the development of people's conceptions of learning from age 5 through 85, and how this kind of knowledge could be used to measure and support learning. In 1998, I received $500,000 from the Spencer Foundation to further develop the methods designed for this research. Some of my areas of expertise are human learning and development, psychometrics, metacognition, moral education, and research methods.
In the simplest possible terms, what I learned in 5 years of graduate school is that the human brain is designed to drive learning, and that preserving that natural drive requires 5 ingredients:
a safe environment that is rich in learning opportunities and healthy human interaction,
a teacher who understands each child's interests and level of tolerance for failure,
a mechanism for determining "what comes next"—what is just challenging enough to allow for success most of the time (but not all of the time),
instant actionable feedback, and
the opportunity to integrate new knowledge or skills into each learner's existing knowledge network well enough to make it useable before pushing instruction to the next level. (We call this building a "robust knowledge network"—the essential foundation for future learning.)*
Identifying the solution
Once we understood what learning should look like, we needed to decide where to intervene. The answer, when it came, was a complete surprise. Understanding what comes next—something that can only be learned by measuring what a student understands now—was an integral part of the recipe for learning. This meant that testing—which we originally saw as an obstacle to robust learning—was actually the solution—but only if we could build tests that would free students to learn the way their brains are designed to learn. These tests would have to help teachers determine "what comes next" (ingredient 3) and provide instant actionable feedback (ingredient 4), while rewarding them for helping students build robust knowledge networks (ingredient 5).
Unfortunately, conventional standardized tests were focused on "correctness" rather than robust learning, and none of them were based on the study of how targeted concepts and skills develop over time. Moreover, they were designed not to support learning, but rather to make decisions about advancement or placement, based on how many correct answers students were able to provide relative to other students. Because this form of testing did not meet the requirements of our learning recipe, we'd have to start from scratch.
Developing the solution
We knew that our solution—reinventing educational testing to serve robust learning—would require many years of research. In fact, we would be committing to possible decades of effort without a guaranteed result. It was the vision of a future educational system in which all children retained their inborn drive for learning that ultimately compelled us to move forward.
To reinvent educational testing, we needed to:
make a deep study of precisely how children build particular knowledge and skills over time in a wide range of subject areas (so these tests could accurately identify "what comes next");
make tests that determine how deeply students understand what they have learned—how well they can use it to address real-world issues or problems (requires that students show how they are thinking, not just what they know—which means written responses with explanations); and
produce formative feedback and resources designed to foster "robust learning" (build robust knowledge networks).
Here's what we had to invent:
A learning ruler (building on Commons  and Fischer );
A method for studying how students learn tested concepts and skills (refining the methods developed for my dissertation);
A human scoring system for determining the level of understanding exhibited in students' written explanations (building upon Commons' and Fischer's methods, refining them until measurements were precise enough for use in educational contexts); and
An electronic scoring system, so feedback and resources could be delivered in real time.
It took over 20 years (1996–2016), but we did it! And while we were doing it, we conducted research. In fact, our assessments have been used in dozens of research projects, including a 25 million dollar study of literacy conducted at Harvard, and numerous Ph.D. dissertations—with more on the way.
What we've learned
We've learned many things from this research. Here are some that took us by surprise:
Students in schools that focus on building deep understanding graduate seniors that are up to 5 years ahead (on our learning ruler) of students in schools that focus on correctness (2.5 to 3 years after taking socioeconomic status into account).
Students in schools that foster robust learning develop faster and continue to develop longer (into adulthood) than students in schools that focus on correctness.
On average, students in schools that foster robust learning produce more coherent and persuasive arguments than students in schools that focus on correctness.
On average, students in our inner-city schools, which are the schools most focused on correctness, stop developing (on our learning ruler) in grade 10.
The average student who graduates from a school that strongly focuses on correctness is likely, in adulthood, to (1) be unable to grasp the complexity and ambiguity of many common situations and problems, (2) lack the mental agility to adapt to changes in society and the workplace, and (3) dislike learning.
From our perspective, these results point to an educational crisis that can best be addressed by allowing students to learn as their brains were designed to learn. Practically speaking, this means providing learners, parents, teachers, and schools with metrics that reward and support teaching that fosters robust learning.
Where we are today
Lectica has created the only metrics that meet all of these requirements. Our mission is to foster greater individual happiness and fulfillment while preparing students to meet 21st century challenges. We do this by creating and delivering learning tools that encourage students to learn the way their brains were designed to learn. And we ensure that students who need our learning tools the most get them first by providing free subscriptions to individual teachers everywhere.
To realize our mission, we organized as a nonprofit. We knew this choice would slow our progress (relative to organizing as a for-profit and welcoming investors), but it was the only way to guarantee that our true mission would not be derailed by other interests.
Thus far, we've funded ourselves with work in the for-profit sector and income from grants. Our background research is rich, our methods are well-established, and our technology works even better than we thought it would. Last fall, we completed a demonstration of our electronic scoring system, CLAS, a novel technology that learns from every single assessment taken in our system.
The groundwork has been laid, and we're ready to scale. All we need is the platform that will deliver the assessments (called DiscoTests), several of which are already in production.
After 20 years of high stakes testing, students and teachers need our solution more than ever. We feel compelled to scale a quickly as possible, so we can begin the process of reinvigorating today's students' natural love of learning, and ensure that the next generation of students never loses theirs. Lectica's story isn't finished. Instead, we find ourselves on the cusp of a new beginning!
A final note: There are many benefits associated with our approach to assessment that were not mentioned here. For example, because the assessment scores are all calibrated to the same learning ruler, students, teachers, and parents can easily track student growth. Even better, our assessments are designed to be taken frequently and to be embedded in low-stakes contexts. For grading purposes, teachers are encouraged to focus on growth over time rather than specific test scores. This way of using assessments pretty much eliminates concerns about cheating. And finally, the electronic scoring system we developed is backed by the world's first "taxonomy of learning," which also serves many other educational and research functions. It's already spawned a developmentally sensitive spell-checker! One day, this taxonomy of learning will be robust enough to empower teachers to create their own formative assessments on the fly.
Adaptive learning technologies are touted as an advance in education and a harbinger of what's to come. But although we at Lectica agree that adaptive learning has a great deal to offer, we have some concerns about its current limitations. In an earlier article, I raised the question of how well one of these platforms, Knewton, serves "robust learning"—the kind of learning that leads to deep understanding and usable knowledge. Here are some more general observations.
The great strength of adaptive learning technologies is that they allow students to learn at their own pace. That's big. It's quite enough to be excited about, even if it changes nothing else about how people learn. But in our excitement about this advance, the educational community is in danger of ignoring important shortcomings of these technologies.
First, adaptive learning technologies are built on adaptive testing technologies. Today, these testing technologies are focused on "correctness." Students are moved to the next level of difficulty based on their ability to get correct answers. This is what today's testing technologies measure best. However, although being able to produce or select correct answers is important, it is not an adequate indication of understanding. And without real understanding, knowledge is not usable and can't be built upon effectively over the long term.
Second, today's adaptive learning technologies are focused on a narrow range of content—the kind of content psychometricians know how to build tests for—mostly math and science (with an awkward nod to literacy). In public education during the last 20 years, we've experienced a gradual narrowing of the curriculum, largely because of high stakes testing and its narrow focus. Today's adaptive learning technologies suffer from the same limitations and are likely to reinforce this trend.
Third, the success of adaptive learning technologies is measured with standardized tests of correctness. Higher scores will help more students get into college—after all, colleges use these tests to decide who will be admitted. But we have no idea how well higher scores on these tests translate into life success. Efforts to demonstrate the relevance of educational practices are few and far between. And notably, there are many examples of highly successful individuals who were poor players in the education game—including several of the worlds' most productive and influential people.
Fourth, some proponents of online adaptive learning believe that it can and should replace (or marginalize) teachers and classrooms. This is concerning. Education is more than a process of accumulating facts. For one thing, it plays an enormous role in socialization. Good teachers and classrooms offer students opportunities to build knowledge while learning how to engage and work with diverse others. Great teachers catalyze optimal learning and engagement by leveraging students' interests, knowledge, skills, and dispositions. They also encourage students to put what they're learning to work in everyday life—both on their own and in collaboration with others.
Lectica has a strong interest in adaptive learning and the technologies that deliver it. We anticipate that over the next few years, our assessment technology will be integrated into adaptive learning platforms to help expand their subject matter and ensure that students are building robust, usable knowledge. We will also be working hard to ensure that these platforms are part of a well-thought out, evidence-based approach to education—one that fosters the development of tomorrow's skills—the full range of skills and knowledge required for success in a complex and rapidly changing world.
There are four keys to optimizing learning and development and ensuring that it continues over a lifetime.
Don’t cram content. Learning doesn’t work optimally when it is rushed or when learners are over-stressed. In Finland, students only go to school three 6-hour days a week, rarely have homework, and do better on PISA than students anywhere else in the world. (Unfortunately, PISA primarily measures correctness, but it’s the best international metric we have at present.) Their educational system is focused on building students’ knowledge networks. Students don’t move on to the next level until they master the current level. The Fins have figured out what our research shows—stuffing content has the long-term effect of slowing or halting development, while a focus on building knowledge networks leads to a steeper learning trajectory and a lifetime of learning and development.
Focus on the network. To learn very large quantities of information, we must effectively recruit System 1 (the fast unconscious brain). System 1 makes associations. (Think of a neural network.) When we learn content through VCoL, we network System 1, connecting new content to already networked content in a way that creates a foundation for what comes next. This does not happen robustly without VCoL, which builds and solidifies the network through application/practice and reflection. System 1 can handle vast amounts of information and processes it rapidly. It serves us well when we learn well.
Make reflection a part of every learning moment. People cannot reason well about things they don’t understand well. When we foster deep understanding through VCoL (and the +7 skills), we recruit System 2 (the slow reasoning brain) to consciously shape the creation and modification of connections in System 1—ensuring that our network of knowledge is growing in a way that mirrors “reality.” The constant practice of analytical and reflective skills not only builds a robust network, but also increases our capacity for making reasonable connections and inferences and enhances our mental agility and capacity for making useful intuitive “leaps.” We learn to think by thinking—and we think better when we have a robust knowledge network to rely on.
Educate the whole person. We believe that education should focus on the development of the entire human being. This means supporting the development of competent, compassionate, aware, and attentive human beings who work well with others. A good way to develop these qualities is through embedded practices that foster interpersonal awareness and skill, such as collaborative or shared learning. These practices provide another benefit as well. They tend to excite emotions that are known to enhance learning.
The best way we know of to accelerate learning is to slow down! It may be counterintuitive, but learning slowly—in ways that foster deep understanding—is the best way to speed up growth! You’ve achieved deep understanding when you’re able to connect new knowledge with your existing knowledge, then put it to work in a variety of real-world contexts.
In an earlier post, I presented evidence that building deep understanding accelerates learning (relative to learning correct answers, rules, definitions, procedures, or vocabulary). In this post, I’m going to explain why.
If you’re a regular reader, you’ll know that my colleagues and I work with a learning model called the” Virtuous Cycle of Learning and +7 skills” (VCoL+7). This model emphasizes the importance of giving learners ample opportunity to build deep understanding through cycles of goal setting, information gathering, application, and reflection. We argue that evidence of deep understanding can be seen in the coherence of people’s arguments—you can’t explain or defend an idea coherently if you don’t understand it—and other evidence of their ability to apply knowledge. When we learn deeply—the VCoL way—we build robust knowledge networks that provide a solid foundation for future learning.
Because poorly understood ideas provide a weak foundation for future learning, my colleagues and I hypothesized that, over time, learners with lower levels of understanding would grow more slowly than learners with higher levels of understanding. We measured understanding by scoring learners’ written arguments for their coherence—how clear and logical they were—on a scale from 1-10. We measured their developmental growth with the Lectical Assessment System, a well-validated developmental scoring system. For details about the study, see the full report.
For the figure below, I’ve borrowed the third graph from the “stop trying to speed up learning” post, which showed growth curves for students in three different kinds of schools. The first (faded teal) group represents students in private schools that emphasized VCoL, the second (faded lime) group represents students in private schools with conventional curricula emphasizing correctness, and the third group (faded red) represents students in public inner city schools with conventional curricula emphasizing correctness. I’ve faded the learning curves for students in these schools into the background.
To this figure, I have added three brightly colored growth curves. These predicted growth curves based on the results of our study on the impact of coherence (which represents level of understanding) on developmental growth. At this point, it’s important to reveal that all of the students included in our study of coherence were students in inner city public schools with a high percentage of students from low income families (the faded red group). Each curve stands for the predicted growth of a hypothetical student from these schools. In the 4th grade, our hypothetical students received time 1 coherence scores of 5.5, 6.5, and 7.5. These values were selected because they were close to the actual time 1 coherence scores for the three groups of students in the background graphic. (Actual average 4th grade scores are shown on the right.) The vertical scale represents developmental level and the horizontal scale represents school grade.
As you can see, the distance between grade 8 scores predicted by the hierarchical regression is a bit less than half of the difference between the actual average scores in the background image. What this means is that in grade 8, almost half of the difference between students in the three types of schools can be explained by depth of understanding (as captured by our measure of coherence).
Both type of instruction and wealth predict learners’ growth trajectories. The results from our study of the impact of coherence on development suggest that if we use forms of instruction that support deep understanding, we can accelerate learning—even for disadvantaged students. These results are consistent with patterns observed in adult learning, in which programs that employ VCoL have been found to accelerate learning relative to programs that emphasize correctness or motivation.
Lectica’s nonprofit mission is to help educators foster deep understanding and lifelong growth. We can do it with your help! Please donate now. Your donation will help us deliver our learning tools—free—to K-12 teachers everywhere.