Honestly folks, we really, really, really need to get over the memorization model of learning. It’s good for spelling bees, trivia games, Jeopardy, and passing multiple choice tests. But it’s BORING if not torturous! And cramming more and more facts into our brains isn’t going to help most of us thrive in real life — especially in the 21st century.
As an employer, I don’t care how many facts are in your head or how quickly you can memorize new information. I’m looking for talent, applied expertise (not just factual or theoretical knowledge), and the following skills and attributes:
The ability to tell the difference between memorizing and understanding
I won’t delegate responsibility to employees who can’t tell the difference between memorizing and understanding. Employees who can’t make this distinction don’t know when they need to ask questions. Consequently, they repeatedly make decisions that aren’t adequately informed.
I’ve taken to asking potential employees what it feels like when they realize they’ve really understood something. Many applicants, including highly educated applicants, don’t understand the question. It’s not their fault. The problem is an educational system that’s way too focused on memorizing.
The ability to think
It’s essential that every employee in my organization is able to evaluate information, solve problems, participate actively in decision making and know the difference between an opinion and a good evidence-based argument.
A desire to listen and the skills for doing it well
We also need employees who want and know how to listen — really listen. In my organization, we don’t make decisions in a vacuum. We seek and incorporate a wide range of stakeholder perspectives. A listening disposition and listening skills are indispensable.
The ability to speak truth (constructively)
I know my organization can’t grow the way I want it to if the people around me are unwilling to share their perspectives or are unable to share them constructively. When I ask someone for an opinion, I want to hear their truth — not what they think I want to hear.
The ability to work effectively with others
This requires respect for other human beings, good interpersonal, collaborative, and conflict resolution skills, the ability to hear and respond positively to productive critique, and buckets of compassion.
Awareness of the ubiquity of human fallibility, including one’s own, and knowledge about human limitations, including the built-in mental biases that so often lead us astray.
A passion for learning (a.k.a. growth mindset)
I love working with people who are driven to increase their understanding and skills — so driven that they’re willing to feel lost at times, so driven that they’re willing to make mistakes on their way to a solution, so driven that their happiness depends on the availability of new challenges.
The desire to do good in the world
I run a nonprofit. We need employees who are motivated to do good.
Not one of these capabilities can be learned by memorizing. All of them are best learned through reflective practice — preferably 12–16 years of reflective practice (a.k.a VCoLing) in an educational system that is not obsessed with remembering.
So, here goes. To keep it simple (or a simple as a discussion of complexity can be), I’m going to limit myself to an exploration of the complexity scores of Presidents Trump (mean score = 1054) and Obama, (mean score = 1163).
If you are unfamiliar with complexity levels, I recommend that you start by watching the short video, below. It provides a general explanation of developmental levels that will help get you oriented.
Adult complexity zones
If you’ve read the previous articles in this series (recommended), you’ve already seen the figure below. It shows the four complexity “zones” that are most common in adulthood and describes them in terms of the kinds of perspectives people performing in each zone are likely to be able to work with effectively. The first zone, advanced linear thinking, is the most common among adults in the United States. It’s also fairly common in the later years of high school—though early linear thinking (not shown here) is more common in that age range.
As development progresses, knowledge and thought move through levels of increasing complexity. Each level builds upon the previous level, which means we have to pass through all of the levels in sequence. Skipping a level is impossible, because a level can’t be built unless there is an earlier level to build upon. As we move through these levels, the evidence of earlier levels does not disappear. It leaves traces in language that can be represented as a kind of history of a person’s development. We call this a developmental profile. To produce a score, CLAS’s algorithm compares an individual’s developmental profile to the typical profiles for each possible score on the complexity scale. Right now, the CLAS algorithm is based on 20 years of rigorous research involving over 45,000 scored interviews, observations, and assessments.
In the second article of this series, I reported that President Trump’s average score (1054) was in the advanced linear thinking zone. Thinking in this zone is abstract and linear. People performing in this zone link ideas in chains of (more or less) logical relations. Reasoning has a “black and white” quality, in the sense that there is a strong preference for simple correct or incorrect answers. Although individuals performing in this level can often see that a situation or problem involves multiple factors, the only way they can organize their thinking about these factors is in chains of logical statements, usually with an “if, then” structure. President Trump, in his interview with The Wall Street Journal on the 25th of July, 2017, provided a typical “if, then” argument when asked about trade with the UK. He argued:
…we’re going to have a very good relationship with the U.K. And we do have to talk to the European Union, because it’s not a reciprocal deal, you know. The word reciprocal, to me, is very important. For instance, we have countries that charge us 100 percent tax to sell a Harley-Davidson into that country. And yet, they sell their motorcycles to us, or their bikes, or anything comparable, and we charge them nothing. There has to be a reciprocal deal. I’m all about that.
The complexity level of an argument can be seen in its structure and the meanings embedded in that structure. This argument has an “if, then” structure, and points to the meaning of reciprocity, which for the President seems to mean an equal exchange—”If you tax at a certain level, then we should tax at that level too.” This kind of “tit for tat” thinking is common in level 10 and below. It’s also a form of thinking that disappears above level 10. For example, in level 11, an individual would be more likely to argue, “It’s more complex than that. There are other considerations that need to be taken into account, like the impact a decision like this is is likely to have on international relations or our citizens’ buying power.” President Trump, in his response, does not even mention additional considerations. This is one of the patterns in his responses that contributed to the score awarded by CLAS.
In the results reported here, a Democrat scored higher than a Republican. We have no reason to believe that conservative thinking is inherently less complex than liberal thinking. In fact, in the past, we have identified highly complex thinking in both conservative and liberal leaders.
A couple of side notes
Upon reading President Trump’s statement above, you may have noticed that, without any framing or context, the President jumped to a discussion of reciprocity. This lack of framing is a ubiquitous feature of President Trump’s arguments. I did not mention it in my discussion of complexity because it is not a direct indicator of thinking complexity. It’s more strongly connected to logical coherence, which correlates with complexity but is not fully explained by complexity.
I’d also like to note that it was difficult to find a single argument in President Trump’s interviews that contained an actual explanation. When asked to explain a position, President Trump was far more likely to (1) tell a story, (2) deride someone, (3) point out his own fame or popularity, or (3) claim that another perspective was a lie or fake news. These were the main ways in which he “backed up” his opinions. Like the absence of framing, these behaviors are not direct indicators of thinking complexity, though they may be correlated with complexity. They are more strongly related to disposition, values, and personality.
These flaws in President Trump’s thinking, combined with the complexity level of his interview responses, should raise considerable alarm. If the President Trump we see is showing us his best thinking—and a casual examination of other examples of his thinking suggests that this is likely to be the case—he clearly lacks the thinking skills demanded by his role. In fact, mid-level management roles generally require better thinking skills than those demonstrated by President Trump.
President Obama’s mean score (1163) was in the advanced systems thinking zone. Thinking in this zone is multivariate and non-linear. People performing in this zone link ideas in complex webs of relations, connecting these webs of relations to one another through common elements. For example, they view individuals as complex webs of traits & behaviors, and groups of individuals as complex webs that include not only the intersections of the webs of their members, but their own distinct properties. Thinking in this zone is very different from thinking in the advanced linear thinking zone. Where individuals performing in the advanced linear thinking zone are concerned about immediate outcomes and proximal causes, individuals performing in the advanced systems thinking zone concern themselves with long term outcomes and systemic causes. Here is an example from President Obama’s interview with the New York Times on March 7th, 2009, in which he explains his approach to economic recovery following the onset of the great recession:
…people have been concerned, understandably, about the decline in the market. Well, the reason the market’s declining is because the economy’s declining and it’s generating a lot of bad news, not surprisingly. And so what I’m focused on is fixing the underlying economy. That’s ultimately what’s going to fix the markets. …in the interim you’ve got some folks who would love to see us artificially prop up the market by just putting in more taxpayer money, which in the short term could make bank balance sheets look better, make creditors and bondholders and shareholders of these financial institutions feel better and you could get a little blip. But we’d be in the exact same spot as we were six, eight, 10 months [ago]. So, what I’ve got to do is make sure that we’re focused on the underlying economy, and … if we do that well …we’re going to get this economy moving again. And I think over the long term we’re going to be much better off.
Rather than offering a pre-determined solution or focusing a single element of the economic crisis, President Obama anchors on the economic system as a system, advocating a comprehensive long-term solution rather than band-aid solutions that might offer some positive immediate results, but would be likely to backfire in the long term. Appreciating that the economic situation presents “a very complex set of problems,” he employs a decision-making process that is “constantly… guided by evidence, facts, talking through all the best arguments, drawing from all the best perspectives, and then talking the best course of action possible.”
The complexity level of president Obama’s thinking as represented in the press interviews analyzed for our study, is a reasonable fit for high office. Of course, we were not able to determine if his scores in this context represent his full capabilities. An informal examination of some of his written work suggests that the “true” complexity level of his thinking may be even higher.
Thinking complexity is not the only factor that plays a role in a president’s success. As president, Obama experienced both successes and failures, and as is usually the case, it’s difficult to say to what extent his solutions contributed to these successes or failures. But, even in the face of this uncertainty, isn’t it a no brainer that a complex problem that’s adequately understood is more likely to be resolved than a complex problem that’s not even recognized?
In his interview with the Wall Street Journal, President Trump claimed that Barack Obama, “didn’t know what the hell he was doing.” Our results suggest that it may be President Trump who doesn’t know what Obama was doing.
An ideal educational assessment strategy—represented above in the assessment triangle—includes three indicators of learning—correctness (content knowledge), complexity (developmental level of understanding), and coherence (quality of argumentation). Lectical Assessments focus primarily on two areas of the triangle—complexity and coherence. Complexity is measured with the Lectical Assessment System, and coherence is measured with a set of argumentation rubrics focused on mechanics, logic, and persuasiveness. We do not focus on correctness, primarily because most assessments already target correctness.
At the center of the assessment triangle is a hazy area. This represents the Goldilocks Zone—the range in which the difficulty of learning tasks is just right for a particular student. To diagnose the Goldilocks Zone, educators evaluate correctness, coherence, and complexity, plus a given learner’s level of interest and tolerance for failure.
When educators work with Lectical Assessments, they use the assessment triangle to diagnose students’ learning needs. Here are some examples:
Level of skill (low, average, high) relative to expectations
This student has relatively high complexity and correctness scores, but his performance is low in coherence. Because lower coherence scores suggest that he has not yet fully integrated his existing knowledge, he is likely to benefit most from participating in interesting activities that require applying existing knowledge in relevant contexts (using VCoL).
This student’s scores are high relative to expectations. Her knowledge appears to be well integrated, but the low correctness suggests that there are gaps in her content knowledge relative to targeted content. Here, we would suggest filling in the missing content knowledge in a way that engages the learner and allows her to integrate it into her well-developed knowledge network.
The scores received by this student are high for correctness, while they are low for complexity and coherence. This pattern suggests that the student is memorizing content without integrating it effectively into his or her knowledge network—and may have been doing this for some time. This student is most likely to benefit from applying their existing content knowledge in personally relevant contexts (using VCoL) until their coherence and complexity scores catch up with their correctness scores.
The scores received by this student are high for correctness, complexity, and coherence. This pattern suggests that the student has a high level of proficiency. Here, we would suggest introducing new knowledge that’s just challenging enough to keep her in her personal Goldilocks zone.
The assessment triangle helps educators optimize learning by ensuring that students are always learning in the Goldilocks Zone. This is a good thing, because students who spend more time in the Goldilocks Zone not only enjoy learning more, they learn better and faster.
The CLAS demo assessment (the LRJA) is a measure of the developmental level of people's reasoning about knowledge, evidence, deliberation, and conflict. People who score higher on this scale are able to work effectively with increasingly complex information and solve increasingly complex problems.
CLAS is the name of our scoring system—the Computerized Lectical Assessment System. It measures the developmental level (hierarchical complexity) of responses on a scale called the Lectical Scale (also called the skill scale).
These dimensions of performance are related to Lectical Level, but they are not the same thing.
The reliability of the CLAS score
The Lectical Scores on CLAS demo assessments are awarded with our electronic scoring system, CLAS.
CLAS scores agree with human scores within 1/5 of a level about 90% of the time. That's the same level of agreement we expect between human raters. This level of agreement is more than acceptable for formative classroom use and program evaluation. It is not good enough for making high stakes decisions.
We don't recommend making high stakes decisions based on the results of any one assessment. Performance over time (growth trajectory) is much more reliable than an individual score.
CLAS is not as well calibrated above 11.5 as it is at lower levels. This is because there are fewer people in our database who perform at the highest levels. As our database grows, CLAS will get better at scoring those performances.
The figure below shows growth curves for four different kinds of K-12 schools in our database. If you want to see how an individual student's growth relates to this graph, we suggest taking at least three assessments over the course of a year or more. (The top performing school "Rainbow," is the Rainbow Community School, in North Carolina.)
Lectica's learning model, VCoL+7, emphasizes the importance of giving students ample opportunity to build well-connected knowledge networks through application and reflection. We argue that evidence of the level of integration in students' knowledge networks can be seen in the quality of their argumentation. In other words, we think of poor arguments as a symptom of poor integration. In the research reported in the video below, we asked if students' ability to make good arguments predicts their rate of growth on the Lectical Scale.