Founder and Executive Director of Lectica, Inc. and founder of the DiscoTest initiative. Dr. Dawson is an award-winning scholar, researcher, educator, and test developer. She has been studying how people learn and how people think about learning for over two decades. Her dissertation, which explored the way people of different ages conceptualize education, learning, testing, and teaching introduced a new set of methods for documenting learning sequences. This work, along with her studies in psychometrics, have provided the basis for a new model of assessment—one that focuses on helping teachers identify the learning needs of individual students. Through the DiscoTest initiative, Dawson and her colleagues have shown that it is possible to design standardized educational assessments that not only help teachers identify the learning needs of individual students, but turn the testing experience into a rich learning experience in which students practice their thinking, communication, and evaluation skills. Scholarly articles by Dawson can be found on the articles page of the <a href="https://dts.lectica.org/_about/articles.php"Lectica site.
There’s a battle out there no one’s tweeting about. It involves a tension between statistical significance and practical significance. If you make decisions that involve evaluating evidence—in other words, if you are human—understanding the distinction between these two types of significance will significantly improve your decisions (both practically and statistically).
Statistical significance (a.k.a. “p”) is a calculation made to determine how confident we can be that a relationship between two factors (variables) is real. The lower a p value, the more confident we can be. Most of the time, we want p to be less than .05.
Don’t be misled! A low p value tells us nothing about the size of a relationship between two variables. When someone says that statistical significance is high, all this means is that we can be more confident that the relationship is real.
Once we know we can be confident that a relationship between two variables is real, we should check to see if the research has been replicated. That’s because we can’t be sure a statistically significant relationship found in a single study is really real. After we’ve determined that a relationship is statistically significant and replicable, it’s time to consider practical significance. Practical significance has to do with the size of the relationship.
To figure out how practically significant a relationship is, we need to know how big it is. The size of a relationship, or effect size, is evaluated independently of p. For a plain English discussion of effect size, check out this article, Statistics for all: prediction.
The greater the size of a relationship between two variables, the more likely the relationship is to be important — but that’s not enough. To have real importance, a relationship must also matter. And it is the decision-maker who decides what matters.
Let’s look at one of my favorite examples. The results of high stakes tests like the SAT and GRE — college entrance exams made by ETS — have been shown to predict college success. Effect sizes tend to be small, but the effects are statistically significant — we can have confidence that they are real. And evidence for these effects have come from numerous studies, so we know they are really real.
If you’re the president of a college, there is little doubt that these test scores have practical significance. Improving prediction of student success, even a little, can have a big impact on the bottom line.
If you’re an employer, you’re more likely to care about how well a student did in college than how they did prior to college, so SAT and GRE scores are likely to be less important to you than college success.
If you’re a student, the size of the effect isn’t important at all. You don’t make the decision about whether or not the school is going to use the SAT or GRE to filter students. Whether or not these assessments are used is out of your control. What’s important to you is how a given college is likely to benefit you.
If you’re me, the size of the effect isn’t very important either. My perspective is that of someone who wants to see major changes in the educational system. I don’t think we’re doing our students any favors by focusing on the kind of learning that can be measured by tests like the GRE and SAT. I think our entire educational system leans toward the wrong goal—transmitting more and more “correct” information. I think we need to ask if what students are learning in school is preparing them for life.
Another thing to consider when evaluating practical significance is whether or not a relationship between two variables tells us only part of a more complex story. For example, the relationship between ethnicity and the rate of developmental growth (what my colleagues and I specialize in measuring) is highly statistically significant (real) and fairly strong (moderate effect size). But, this relationship completely disappears once socioeconomic status (wealth) is taken into account. The first relationship is misleading (spurious). The real culprit is poverty. It’s a social problem, not an ethnic problem.
Most discussions of practical significance stop with effect size. From a statistical perspective, this makes sense. Statistics can’t be used to determine which outcomes matter. People have to do that part, but statistics, when good ones are available, should come first. Here’s my recipe:
My organization, Lectica, Inc., is a 501(c)3 nonprofit corporation. Part of our mission is to share what we learn with the world. One of the things we’ve learned is that many assessment buyers don’t seem to know enough about statistics to make the best choices. The Statistics for all series is designed to provide assessment buyers with the knowledge they need most to become better assessment shoppers.
Why you might want to reconsider using 360s and EQ assessments to predict recruitment success
Measurements are often used to make predictions. For example, they can help predict how tall a 4-year-old is likely to be in adulthood, which students are likely to do better in an academic program, or which candidates are most likely to succeed in a particular job.
Some of the attributes we measure are strong predictors, others are weaker. For example, a child’s height at age 4 is a pretty strong predictor of adult height. Parental height is a weaker predictor. The complexity of a person’s workplace decision making, on its own, is a moderate predictor of success in the workplace. But the relation between the complexly of their workplace decision making and the complexity of their role is a strong predictor.
How do we determine the strength or a predictor? In statistics, the strength of predictions is represented by an effect size. Most effect size indicators are expressed as decimals and range from .00 –1.00, with 1.00 representing 100% accuracy of prediction. The effect size indicator you’ll see most often is r-square. If you’ve ever been forced to take a statistics course—;)—you may remember that r represents the strength of a correlation. Before I explain r-square, let’s look at some correlation data.
The four figures below represent 4 different correlations, from weakest (.30) to strongest (.90). Let’s say the vertical axis (40 –140) represents the level of success in college, and the horizontal axis (50 –150) represents scores on one of 4 college entrance exams. The dots represent students. If you were trying to predict success in college, you would be wise to choose the college entrance exam that delivered an r of .90.
Why is an r of .90 preferable? Well, take a look at the next set of figures. I’ve drawn lines through the clouds of dots (students) to show regression lines. These lines represent the prediction we would make about how successful a student will be, given a particular score. It’s clear that in the case of the first figure (r =.30), this prediction is likely to be pretty inaccurate. Many students perform better or worse than predicted by the regression line. But as the correlations increase in size, prediction improves. In the case of the fourth figure (r =.90), the prediction is most accurate.
What does a .90 correlation mean in practical terms? That’s where r-square comes in. If we multiply .90 by .90 (calculate the square), we get an r-square of .81. Statisticians would say that the predictor (test score), explains 81% of the variance in college success. The 19% of the variance that’s not explained (1.00 -.81 =.19) represents the percent of the variance that is due to error (unexplained variance). The square root of 19% is the amount of error (.44).
Correlations of .90 are very rare in the social sciences—but even correlations this strong are associated with a significant amount of error. It’s important to keep error in mind when we use tests to make big decisions—like who gets hired or who gets to go to college. When we use tests to make decisions like these, the business or school is likely to benefit—slightly better prediction can result in much better returns. But there are always rejected individuals who would have performed well, and there are always accepted individuals who will perform badly.
Let’s get realistic. As I mentioned earlier, correlations of .90 are very rare. In recruitment contexts, the most predictive assessments (shown above) correlate with hire success in the range of .50 –.54, predicting from 25% – 29% of the variance in hire success. That leaves a whopping 71% – 75% of the variance unexplained, which is why the best hiring processes not only use the most predictive assessments, but also consider multiple predictive criteria.
On the other end of the spectrum, there are several common forms of assessment that explain less than 9% of the variance in recruitment success. Their correlations with recruitment success are lower than .30. Yet some of these, like 360s, reference checks, and EQ, are wildly popular. In the context of hiring, the size of the variance explained by error in these cases (more than 91%) means there is a very big risk of being unfair to a large percentage of candidates. (I’m pretty certain assessment buyers aren’t intentionally being unfair. They probably just don’t know about effect size.)
If you’ve read my earlier article about replication, you know that the power-posing research could not be replicated. You also might be interested to learn that the correlations reported in the original research were also lower than .30. If power-posing had turned out to be a proven predictor of presentation quality, the question I’d be asking myself is, “How much effort am I willing to put into power-posing when the variance explained is lower than 9%?”
If we were talking about something other than power-posing, like reducing even a small risk that my child would die of a contagious disease, I probably wouldn’t hesitate to make a big effort. But I’m not so sure about power-posing before a presentation. Practicing my presentation or getting feedback might be a better use of my time.
Summing up (for now)
A basic understanding of prediction is worth cultivating. And it’s pretty simple. You don’t even have to do any fancy calculations. Most importantly, it can save you time and tons of wasted effort by giving you a quick way to estimate the likelihood that an activity is worth doing (or product is worth having). Heck, it can even increase fairness. What’s not to like?
My organization, Lectica, Inc., is a 501(c)3 nonprofit corporation. Part of our mission is to share what we learn with the world. One of the things we’ve learned is that many assessment buyers don’t seem to know enough about statistics to make the best choices. The Statistics for all series is designed to provide assessment buyers with the knowledge they need most to become better assessment shoppers.
(Why you should have been suspicious of power-posing from the start!)
I’ve got a free, low-tech life hack for you that will save significant time and money — and maybe even improve your health. All you need to do is one little thing. Before you let the latest research results change your behavior, check to see if the research has been replicated!
One of the hallmarks of modern science is the notion that one study of a new phenomenon—especially a single small study—proves nothing. Most of the time, the results of such studies can do little more than suggest possibilities. To arrive at proof, results have to be replicated—again and again, usually in a variety of contexts. This is important, especially in the social sciences, where phenomena are difficult to measure and the results of many new studies cannot be replicated.
Researchers used to be trained to avoid even implying that findings from a new study were proven facts. But when Amy Cuddy set out to share the results of her and her colleagues’ power-posing research, she didn’t simply imply that her results could be generalized. She unabashedly announced to an enthralled Ted Talk audience that she’d discovered a “Free, no-tech life hack…that could significantly change how your life unfolds.”
Thanks to this talk, many thousands—perhaps millions—of people-hours have been spent power-posing. But it’s not the power-posers whose lives have changed. Unfortunately, as it turns out, it’s Dr. Cuddy’s life that changed significantly—when other researchers were unable to replicate her results. In fact, because she had made such strong unwarranted claims, Dr. Cuddy became the focus of severe criticism.
Although she was singled out, Dr. Cuddy is far from alone. She’s got lots of company. Many fads have begun just like Power Posing did. Here’s how it goes: A single small study produces results that have “novelty appeal,” the Today Show picks up the story, and thousands jump on the bandwagon! Sometimes, as in the case of power-posing, the negative impact is no worse than a bit of wasted time. But in other cases, such as when our heath or pocketbooks are at stake, the impacts can be much greater.
“But it worked for me!” If you tried power-posing and believe it was responsible for your success in achieving an important goal, you may be right. The scientific method isn’t perfect — especially in the social sciences — and future studies with better designs may support your belief. However, I recommend caution in relying on personal experience. Humans have powerful built-in mental biases that lead us to conclude that positive outcomes are caused by something we did to induce them. This makes it very difficult for us to distinguish between coincidence and cause. And it’s one reason we need the scientific method, which is designed to help us reduce the impact of these biases.
Replication matters in assessment development, too
Over the last couple of decades, I’ve looked at the reliability & validity evidence for many assessments. The best assessment developers set a pretty high replication standard, conducting several validity & reliability studies for each assessment they offer. But many assessment providers—especially those serving businesses—are much more lax. In fact, many can point to only a single study of reliability and validity. To make matters worse, in some cases, that study has not been peer reviewed.
Be wary of assessments that aren’t backed by several studies of reliability and validity.
In the first post in this series, I promised to share a quick and dirty trick for determining how much confidence you can have in a test score. I will. But first, I want to show you a bit more about what estimating confidence means when it comes to educational and psychological tests.
Let’s start with a look at how test scores are usually reported. The figure below shows three scores, one at level 8, one at level 6, and one at level 4. Looking at this figure, most of us would be inclined to assume that these scores are what they seem to be—precise indicators of the level of a trait or skill.
But this is not the case. Test scores are fuzzy. They’re best understood as ranges ratherthan as points on a ruler. In other words, test scores are always surrounded by confidence intervals. A person’s true score is likely to fall somewhere in the range described by the confidence interval around a test score.
In order to figure out how fuzzy a test score actually is, you need one thing—an indicator of statistical reliability. Most of the time, this is something called Cronbach’s Alpha. All good test developers publish information about the statistical reliability of their measures, ideally in refereed academic journals with easy to find links on their web sites! If a test developer won’t provide you with information about Alpha (or its equivalent) for each score reported on a test, it’s best to move on.
The higher the reliability (usually Alpha) the smaller the confidence interval. And the smaller the confidence interval, the more confidence you can have in a test score.
The table below will help to clarify why it is important to know Alpha (or its equivalent). It shows the relationship between Alpha (which can range from 0 to 1.0) and the number of distinct levels (strata) a test can be said to have. For example, an assessment with a reliability of .80, has 3 strata, whereas an assessment with a reliability of .94 has 5.
Strata have direct implications for the confidence we can have in a person’s score on a given assessment, because they tell us about the range within which a person’s true score would fall—its confidence interval—given the score awarded.
Imagine that you have just taken a test of emotional intelligence with a score range of 1 to 10 and a reliability of .95. The number of strata into which an assessment with a reliability of .95 can be divided is about 6, which means that each strata equals about 1.75 points on the 10 point scale (10 divided by 6). If your score on this test was 8, your true score would likely be somewhere between 7.13 and 8.88—your score’s confidence interval.
The figure below shows the true score ranges for three test takers, CB, RM, and PR. The fact that these ranges don’t overlap gives us confidence that the emotional intelligence of these test-takers is actually different**.
If these scores were closer together, their confidence intervals would overlap. And if that was the case—for example if you were comparing two individuals with scores of 8 and 8.5—it would not be correct to say the scores were different form one another. In fact, it would be incorrect for a hiring manager to consider the difference between a score of 8 and a score of 8.5 in making a choice between two job candidates.
By the way, tests with Alphas in the range of .94 or higher are considered suitable for high-stakes use (assuming that they meet other essential validity requirements). What you see in the figure below is about as good as it gets in educational and psychological assessment.
Most assessments used in organizations do not have Alphas that are anywhere near .95. Some of the better assessments have Alphas as high as .85. Let’s take a look at what an Alpha at this level does to confidence intervals.
If the test you have taken has a score range of 1–10 and an Alpha (reliability) of .85, the number of strata into which this assessment can be divided is about 3.4, which means that each strata equals about 2.9 (10 divided by 3.4) points on the 10 point scale. In this case, if you receive a score of 8, your true score is likely to fall within the range of 6.6 to 9.5*.
In the figure below, note that CB’s true score range now overlaps RM’s true score range and RM’s true score range overlaps PR’s true score range. This means we cannot say—with confidence—that CB’s score is different from RM’s score, or that RM’s score is different from PR’s score.
Assessments with Alphas in the .85 range are suitable for classroom use or low-stakes contexts. Yet, every day, schools and businesses use tests with reliabilities in the .85 range to make high stakes decisions—such as who will be selected for advancement or promotion. And this is often done in a way that would exclude RM (yellow circle) even though his confidence interval overlaps CB’s (teal circle) confidence interval.
Many tests used in organizations have Alphas in the .75 range. If the test you have taken has a score range of 1–10 and an Alpha of .75, the number of strata into which this assessment can be divided is about 2.2, which means that each strata equals about 4.5 points on the 10 point scale. In this case, if you receive a score of 8, your true score is likely to fall within the range of 6–10*.
As shown in the figure below, scores would now have to differ by at least 4.5 points in order for us to distinguish between two people. CB’s and PR’s scores are different, but RM’s score is uninterpretable.
Tests or sub-scales with alphas in the .75 range are considered suitable for research purposes. Yet, sad to say, schools and businesses now use tests with scales or sub-scales that have Alphas in or below the .75 range, treating these scores as if they provide useful information, when in most cases the scores—like RM’s—are uninterpretable.
If your current test providers are not reporting true score ranges (confidence intervals), ask for them. If they only provide Alphas (reliability statistics) you can use the table and figures in this article to calculate true score ranges for yourself. If you don’t want to do the math, no problem. You can use the figures above to get a feel for how precise a score is.
Statistical reliability is only one of the ways in which assessments should be evaluated. Test developers should also ask how well an assessment measures what it is intended to measure. And those who use an assessment should ask whether or not what it measures is relevant or important. I’ll be sharing some tricks for looking at these forms of validity in future articles.
What is complexity level? In my work, a complexity level is a point or range on a dimension called hierarchical complexity. In this article, I’m not going to explain hierarchical complexity, but I am going to try to illustrate—in plain(er) English—how complexity level relates to decision-making skills, workplace roles, and curricula. If you’re looking for a more scholarly definition, you can find it in our academic publications. The Shape of Development is a good place to begin.
My colleagues and I make written-response developmental assessments that are designed to support optimal learning and development. All of these assessments are scored for their complexity level on a developmental scale called the Lectical Scale. It’s a scale of increasing hierarchical complexity, with 13 complexity levels (0–12) that span birth through adulthood. On this scale, each level represents a way of seeing the world. Each new level builds upon the previous level, so thinking in a new complexity level is more complex and abstract than thinking at the precious level. The following video describes levels 5–12.
We have five ways of representing Lectical Level scores, depending on the context: (1) as whole levels (9, 10, 11, etc.), (2) as decimals (10.35, 11.13, etc.), (3) as 4 digit numbers (1035, 1113, etc.), (4) as 1/4 of a level phase scores (10a, 10b, 10c, 10d, 11a, etc.), and (5) as 1/2 of a level zone scores (early level 10, advanced level 10; early level 11, etc.).
Interpreting Lectical (complexity level) Scores
Lectical Scores are best thought of in terms of the specific skills, meanings, tasks, roles, or curricula associated with them. To illustrate, I’m including table below that shows…
Lectical Score ranges for the typical complexity of coursework and workplace roles (Role demands & Complexity demands), and
some examples of decision making skills demonstrated in these Lectical Score ranges.
In the last bullet above, I highlighted the term skill, because we differentiate between skills and knowledge. Lectical Scores don’t represent what people know, they represent the complexity of the skill used to apply what they know in the real world. This is important, because there’s a big difference between committing something to memory and understanding it well enough to put it to work. For example, in the 1140–1190 range, the first skill mentioned in the table below is the “ability to identify multiple relations between nested variables.” The Lectical range in this row does not represent the range in which people are able to make this statement. Instead, it represents the level of complexity associated with actually identifying multiple relations between nested variables.
If you want to use this table to get an idea of how skills increase in complexity over time, I suggest that you begin by comparing skill descriptions in ranges that are far apart. For example, try comparing the skill description in the 945–995 range with the skill descriptions in the 1250–1300 range. The difference will be obvious. Then, work your way toward closer and closer ranges. It’s not unusual to have difficulty appreciating the difference between adjacent ranges—that generally takes time and training—but you’ll find it easy to see differences that are further apart.
When using this table as a reference, please keep in mind that several factors play a role in the actual complexity demands of both coursework and roles. In organizations, size and sector matter. For example, there can be a difference as large as 1/2 of a level between freshman curricula in different colleges.
I hope you find this table helpful (even though it’s difficult to read). I’ll be using it as a reference in future articles exploring some of what my colleagues and I have learned by measuring and studying complexity level—starting with leader decision-making.
So, here goes. To keep it simple (or a simple as a discussion of complexity can be), I’m going to limit myself to an exploration of the complexity scores of Presidents Trump (mean score = 1054) and Obama, (mean score = 1163).
If you are unfamiliar with complexity levels, I recommend that you start by watching the short video, below. It provides a general explanation of developmental levels that will help get you oriented.
Adult complexity zones
If you’ve read the previous articles in this series (recommended), you’ve already seen the figure below. It shows the four complexity “zones” that are most common in adulthood and describes them in terms of the kinds of perspectives people performing in each zone are likely to be able to work with effectively. The first zone, advanced linear thinking, is the most common among adults in the United States. It’s also fairly common in the later years of high school—though early linear thinking (not shown here) is more common in that age range.
As development progresses, knowledge and thought move through levels of increasing complexity. Each level builds upon the previous level, which means we have to pass through all of the levels in sequence. Skipping a level is impossible, because a level can’t be built unless there is an earlier level to build upon. As we move through these levels, the evidence of earlier levels does not disappear. It leaves traces in language that can be represented as a kind of history of a person’s development. We call this a developmental profile. To produce a score, CLAS’s algorithm compares an individual’s developmental profile to the typical profiles for each possible score on the complexity scale. Right now, the CLAS algorithm is based on 20 years of rigorous research involving over 45,000 scored interviews, observations, and assessments.
In the second article of this series, I reported that President Trump’s average score (1054) was in the advanced linear thinking zone. Thinking in this zone is abstract and linear. People performing in this zone link ideas in chains of (more or less) logical relations. Reasoning has a “black and white” quality, in the sense that there is a strong preference for simple correct or incorrect answers. Although individuals performing in this level can often see that a situation or problem involves multiple factors, the only way they can organize their thinking about these factors is in chains of logical statements, usually with an “if, then” structure. President Trump, in his interview with The Wall Street Journal on the 25th of July, 2017, provided a typical “if, then” argument when asked about trade with the UK. He argued:
…we’re going to have a very good relationship with the U.K. And we do have to talk to the European Union, because it’s not a reciprocal deal, you know. The word reciprocal, to me, is very important. For instance, we have countries that charge us 100 percent tax to sell a Harley-Davidson into that country. And yet, they sell their motorcycles to us, or their bikes, or anything comparable, and we charge them nothing. There has to be a reciprocal deal. I’m all about that.
The complexity level of an argument can be seen in its structure and the meanings embedded in that structure. This argument has an “if, then” structure, and points to the meaning of reciprocity, which for the President seems to mean an equal exchange—”If you tax at a certain level, then we should tax at that level too.” This kind of “tit for tat” thinking is common in level 10 and below. It’s also a form of thinking that disappears above level 10. For example, in level 11, an individual would be more likely to argue, “It’s more complex than that. There are other considerations that need to be taken into account, like the impact a decision like this is is likely to have on international relations or our citizens’ buying power.” President Trump, in his response, does not even mention additional considerations. This is one of the patterns in his responses that contributed to the score awarded by CLAS.
In the results reported here, a Democrat scored higher than a Republican. We have no reason to believe that conservative thinking is inherently less complex than liberal thinking. In fact, in the past, we have identified highly complex thinking in both conservative and liberal leaders.
A couple of side notes
Upon reading President Trump’s statement above, you may have noticed that, without any framing or context, the President jumped to a discussion of reciprocity. This lack of framing is a ubiquitous feature of President Trump’s arguments. I did not mention it in my discussion of complexity because it is not a direct indicator of thinking complexity. It’s more strongly connected to logical coherence, which correlates with complexity but is not fully explained by complexity.
I’d also like to note that it was difficult to find a single argument in President Trump’s interviews that contained an actual explanation. When asked to explain a position, President Trump was far more likely to (1) tell a story, (2) deride someone, (3) point out his own fame or popularity, or (3) claim that another perspective was a lie or fake news. These were the main ways in which he “backed up” his opinions. Like the absence of framing, these behaviors are not direct indicators of thinking complexity, though they may be correlated with complexity. They are more strongly related to disposition, values, and personality.
These flaws in President Trump’s thinking, combined with the complexity level of his interview responses, should raise considerable alarm. If the President Trump we see is showing us his best thinking—and a casual examination of other examples of his thinking suggests that this is likely to be the case—he clearly lacks the thinking skills demanded by his role. In fact, mid-level management roles generally require better thinking skills than those demonstrated by President Trump.
President Obama’s mean score (1163) was in the advanced systems thinking zone. Thinking in this zone is multivariate and non-linear. People performing in this zone link ideas in complex webs of relations, connecting these webs of relations to one another through common elements. For example, they view individuals as complex webs of traits & behaviors, and groups of individuals as complex webs that include not only the intersections of the webs of their members, but their own distinct properties. Thinking in this zone is very different from thinking in the advanced linear thinking zone. Where individuals performing in the advanced linear thinking zone are concerned about immediate outcomes and proximal causes, individuals performing in the advanced systems thinking zone concern themselves with long term outcomes and systemic causes. Here is an example from President Obama’s interview with the New York Times on March 7th, 2009, in which he explains his approach to economic recovery following the onset of the great recession:
…people have been concerned, understandably, about the decline in the market. Well, the reason the market’s declining is because the economy’s declining and it’s generating a lot of bad news, not surprisingly. And so what I’m focused on is fixing the underlying economy. That’s ultimately what’s going to fix the markets. …in the interim you’ve got some folks who would love to see us artificially prop up the market by just putting in more taxpayer money, which in the short term could make bank balance sheets look better, make creditors and bondholders and shareholders of these financial institutions feel better and you could get a little blip. But we’d be in the exact same spot as we were six, eight, 10 months [ago]. So, what I’ve got to do is make sure that we’re focused on the underlying economy, and … if we do that well …we’re going to get this economy moving again. And I think over the long term we’re going to be much better off.
Rather than offering a pre-determined solution or focusing a single element of the economic crisis, President Obama anchors on the economic system as a system, advocating a comprehensive long-term solution rather than band-aid solutions that might offer some positive immediate results, but would be likely to backfire in the long term. Appreciating that the economic situation presents “a very complex set of problems,” he employs a decision-making process that is “constantly… guided by evidence, facts, talking through all the best arguments, drawing from all the best perspectives, and then talking the best course of action possible.”
The complexity level of president Obama’s thinking as represented in the press interviews analyzed for our study, is a reasonable fit for high office. Of course, we were not able to determine if his scores in this context represent his full capabilities. An informal examination of some of his written work suggests that the “true” complexity level of his thinking may be even higher.
Thinking complexity is not the only factor that plays a role in a president’s success. As president, Obama experienced both successes and failures, and as is usually the case, it’s difficult to say to what extent his solutions contributed to these successes or failures. But, even in the face of this uncertainty, isn’t it a no brainer that a complex problem that’s adequately understood is more likely to be resolved than a complex problem that’s not even recognized?
In his interview with the Wall Street Journal, President Trump claimed that Barack Obama, “didn’t know what the hell he was doing.” Our results suggest that it may be President Trump who doesn’t know what Obama was doing.
How well does the thinking of recent US Presidents stand up to the complexity of issues faced in their role?Special thanks to my Australian colleague, Aiden M. A. Thornton, PhD. Cand., for his editorial and research assistance.
This is the second in a series of articles on the complexity of national leaders’ thinking, as measured with CLAS, a newly validated electronic developmental scoring system. This article will make more sense if you begin with the first article in the series.
Just in case you choose not to read or revisit the first article, here are a few things to keep in mind.
The complexity level of leaders’ thinking is one of the strongest predictors of leader advancement and success.
Many of the issues faced by national leaders require principles thinking (level 12 on the skill scale, illustrated in the figure below).
To accurately measure the complexity level of someone’s thinking (on a given topic), we need examples of their best thinking. In this case, that kind of evidence wasn’t available. As an alternative, my colleagues and I have chosen to examine the complexity level of Presidents’ responses to interviews with prominent journalists.
In this article, we examine the thinking of the four most recent Presidents of the United States — Bill Clinton, George W. Bush, Barack Obama, and Donald Trump. For each president, we selected 3 interviews, based on the following criteria: They
were conducted by prominent journalists representing respected news media;
included questions that requested explanations of the president’s perspective; and
were either conducted within the president’s first year in office or were the earliest interviews we could locate that met the first two criteria.
As noted in the introductory article of this series, we do not imagine that the responses provided in these interviews necessarily represent competence. It is common knowledge* that presidents and other leaders typically attempt to tailor messages for their audiences, so even when responding to interview questions, they may not show off their own best thinking.
Media also tailor writing for their audiences, so to get a sense of what a typical complexity level target for top media might be, we used CLAS to score 11 articles on topics similar to those discussed by the four presidents in their interviews. We selected these articles at random — literally selecting the first ones that came to hand — from recent issues of the New York Times, Guardian, Washington Post, and Wall Street Journal. Articles from all of these newspapers landed in the middle range of the early systems thinking zone, with an average score of 1124.
Based on this information, and understanding that presidents generally attempt to tailor messages for their audience, we hypothesized that presidents would aim for a similar range.
The results were mixed. Only Presidents Clinton and Bush consistently performed in the anticipated range. President Trump stood out by performing well-below this range. His scores were all identical — and roughly equivalent to the average for 12th graders in a reasonably good high school. President Obama also missed the mark, but in the opposite direction. In his first interviews, he scored at the top of the advanced systems thinking zone. But he didn’t stay there. By the time of September’s interview, he was responding in the early systems thinking zone. He even mentioned simplifying communication in this interview. Commenting on his messaging around health care, he said, “I’ve tried to keep it digestible… it’s very hard for people to get… their whole arms around it.”
The Table below shows the complexity scores received by our four presidents. (All of the interviews can readily be found in the presidential archives.)
In the first article of this series, I discussed the importance of attempting to “hire” leaders whose complexity level scores are a good match for the complexity level of the issues they face in their roles. I then posed two questions:
When asked by prominent journalists to explain their positions on complex issues, what is the average complexity level of national leaders’ responses?
How does the complexity level of national leaders’ responses relate to the complexity of the issues they discuss?”
The answer to question 1 is that the average complexity level of presidents’ responses to interview questions varied dramatically. President Trump’s average complexity level score was 1054 — near the average score received by 12th graders in a good high school. President Bush’s average score was 1107 — near the average score received by entry- to mid-level managers in a large corporation. President Clinton’s average score was 1141, near the average score received by upper level managers in large corporations. Obama’s, average score was 1163 — near the the average score of senior leaders in large corporations. (Obama’s highest scores were closer to the average for CEOs in our database.)
With respect to question 2, the complexity level of presidents’ responses did not rise to the complexity level of many of the issues raised in their interviews. These issues ranged from international relations and the economy to health care and global warming. All of these are thorny problems involving multiple interacting and nested systems—early principles and above. Indeed, many of these problems are so complex that they are beyond the capability of even the most complex thinkers to fully grasp. (See my article on the Complexity Gap for more on this issue.) President Obama came closest to demonstrating a level of thinking complexity that would be adequate for coping with problems of this kind. (For more on this, see the third article in this series, If a U. S. President thought like a teenager…)
Obama also demonstrated some of the other qualities required for working well with complexity, such as skills for perspective seeking and perspective coordination, and familiarity with tools for working with complexity—but that’s another story.
In addition to addressing the two questions posed in the first article of this series, we were able to ask if these U. S. presidents seemed to tailor the complexity level of their interview responses for the audiences of the media outlets represented by journalists conducting the interviews.
First, the responses of presidents Bush and Clinton were in the same zone as a set of articles collected from these media outlets. Of course, we can’t be sure the alignment was intentional. There are other plausible explanations, including the possibility that what we witnessed was their best thinking.
In contrast, however, President Trump’s responses were well below the zone of the selected articles, making it difficult to argue that he was tailoring his responses for their audiences. Individuals whose thinking is complex are likely to find thinking at lower levels of complexity simplistic and unsatisfying. Delivering a message that is likely to lead to judgments of this kind does not seem like a rational tactic — especially for a politician.
It seems more plausible that President Trump was demonstrating his best thinking about the issues raised in his interviews. If so, his best would be far below the complexity level of most issues faced in his role. Indeed, individuals performing in the advanced linear thinking zone would not even be aware of the complexity inherent in many of the issues faced daily by national leaders.
President Obama confronted a different challenge. The complexity of thinking evident in his early interviews was very high. Even though, as with Bush and Clinton, it isn’t possible to say we witnessed Obama’s best thinking, we would argue that what we saw of President Obama’s thinking in his first two interviews was a reasonable fit to the complexity of the challenges in his role. However, it appears that Obama soon learned that in order to communicate effectively with citizens, he needed to make his communications more accessible.
In the results reported here, Democrats scored higher than Republicans. We have no reason to believe that conservative thinking is inherently less complex than liberal thinking. In fact, in the past, we have identified highly complex thinking in both conservative and liberal leaders.
We need leaders who can cope with highly complex issues, and particularly in a democracy, we also need leaders we can understand. President Obama showed himself to be a complex thinker, but he struggled with making his communications accessible. President Trump’s message is accessible, but our results suggest that he may not even be aware of the complexity of many issues faced in his role. Is it inevitable that the tension between complexity and accessibility will sometimes lead us to “hire” national leaders who are easy to understand, but lack the ability to work with complexity? And how can we even know if a leader is equipped with the thinking complexity that’s required if candidates routinely simplify communications for their audience? Given our increasingly volatile and complex world, these are questions that cry out for answers.
We don’t have these answers, and we’ve intentionally resisted going deeper into the implications of these findings. Instead, we’re hoping to stimulate discussion around our questions and the implications that arise from the findings presented here. Please feel free to chime in or contact us to further the conversation. And stay tuned. The Australian Prime Ministers are next!
*The speeches of presidents are generally written to be accessible to a middle school audience. The metrics used to determine reading level are not measures of complexity level, but reading level scores are moderately correlated with complexity level.
Special thanks to my Australian colleague, Aiden Thornton, for his editorial and research assistance.
This is the first in a series of articles on the complexity of national leaders’ thinking. These articles will report results from research conducted with CLAS, our newly validated electronic developmental scoring system. CLAS will be used to score these leaders’ responses to questions posed by prominent journalists.
In this first article, I’ll be providing some of the context for this project, including information about how my colleagues and I think about complexity and its role in leadership. I’ve embedded lots of links to additional material for readers who have questions about our 100+ year-old research tradition, Lectica’s (the nonprofit that owns me) assessments, and other research we’ve conducted with these assessments.
Context and research questions
Lectica creates diagnostic assessments for learningthat support the development of mental skills required for working with complexity. We make these learning tools for both adults and children. Our K-12 initiative—the DiscoTest Initiative—is dedicated to bringing these tools to individual K-12 teachers everywhere, free of charge. Its adult assessments are used by organizations in recruitment and training, and by colleges and universities in admissions and program evaluation.
All Lectical Assessments measure the complexity level (aka, level of vertical development) of people’s thinking in particular knowledge areas. A complexity level score on a Lectical Assessment tells us the highest level of complexity—in a problem, issue, or task—an individual is likely to be able to work with effectively.
On several occasions over the last 20 years, my colleagues and I have been asked to evaluate the complexity of national leaders’ reasoning skills. Our response has been, “We will, but only when we can score electronically—without the risk of human bias.” That time has come. Now that our electronic developmental scoring system, CLAS, has demonstrated a level of reliability and precision that is acceptable for this purpose, we’re ready to take a look.
Evaluating the complexity of national leaders’ thinking is a challenging task for several reasons. First, it’s virtually impossible to find examples of many of these leaders’ “best work.” Their speeches are generally written for them, and speech writers usually try to keep the complexity level of these speeches low, aiming for a reading level in the 7th to 9th grade range. (Reading level is not the same thing as complexity level, but like most tests of capability, it correlates moderately with complexity level.) Second, even when national leaders respond to unscripted questions from journalists, they work hard to use language that is accessible to a wide audience. And finally, it’s difficult to identify a level playing field—one in which all leaders have the same opportunity to demonstrate the complexity of their thinking.
Given these obstacles, there’s no point in attempting to evaluate the actual thinking capabilities of national leaders. In other words, we won’t be claiming that the scores awarded by CLAS represent the true complexity level of leaders’ thinking. Instead, we will address the following questions:
When asked by prominent journalists to explain their positions on complex issues, what is the average complexity level of national leaders’ responses?
How does the complexity level of national leaders’ responses relate to the complexity of the issues they discuss?
Thinking complexity and leader success
At this point, you may be wondering, “What is thinking complexity and why is it important?” A comprehensive response to this question isn’t possible in a short article like this one, but I can provide a basic description of complexity as we see it at Lectica, and provide some examples that highlight its importance.
All issues faced by leaders are associated with a certain amount of built-in complexity. For example:
The sheer number of factors/stakeholders that must be taken into account.
Short and long-term implications/repercussions. (Will a quick fix cause problems downstream, such as global unrest or catastrophic weather?)
The number and diversity of stakeholders/interest groups. (What is the best way to balance the needs of individuals, families, businesses, communities, states, nations, and the world?)
The length of time it will take to implement a decision. (Will it take months, years, decades? Longer projects are inherently more complex because of changes over time.)
Formal and informal rules/laws that place limits on the deliberative process. (For example, legislative and judicial processes are often designed to limit the decision making powers of presidents or prime ministers. This means that leaders must work across systems to develop decisions, which further increases the complexity of decision making.)
Over the course of childhood and adulthood, the complexity of our thinking develops through up to 13 skill levels (0–12). Each new level builds upon the previous level. The figure above shows four adult complexity “zones” — advanced linear thinking (second zone of level 10), early systems thinking (first zone of level 11), advanced systems thinking (second zone of level 11), early principles thinking (first zone of level 12). In advanced linear thinking, reasoning is often characterized as “black and white.” Individuals performing in this zone cope best with problems that have clear right or wrong answers. It is only once individuals enter early systems thinking, that we begin to work effectively with highly complex problems that do not have clear right or wrong answers.
Leadership at the national level requires exceptional skills for managing complexity, including the ability to deal with the most complex problems faced by humanity (Helbing, 2013). Needless to say, a national leader regularly faces issues at or above early principles thinking.
Complexity level and leadership—the evidence
In the workplace, the hiring managers who decide which individuals will be put in leadership roles are likely to choose leaders whose thinking complexity is a good match for their roles. Even if they have never heard the term complexity level, hiring managers generally understand, at least implicitly, that leaders who can work with the complexity inherent in the issues associated with their roles are likely to make better decisions than leaders whose thinking is less complex.
There is a strong relation between the complexity of leadership roles and the complexity level of leaders’ reasoning. In general, more complex thinkers fill more complex roles. The figure below shows how lower and senior level leaders’ complexity scores are distributed in Lectica’s database. Most senior leaders’ complexity scores are in or above advanced systems thinking, while those of lower level leaders are primarily in early systems thinking.
The strong relation between the complexity of leaders’ thinking and the complexity of their roles can also be seen in the recruitment literature. To be clear, complexity is not the only aspect of leadership decision making that affects leaders’ ability to deal effectively with complex issues. However, a large body of research, spanning over 50 years, suggests that the top predictors of workplace leader recruitment success are those that most strongly relate to thinking skills, including complexity level.
The figure below shows the predictive power of several forms of assessment employed in making hiring and promotion decisions. Assessments of mental ability have been shown to have the highest predictive power. In other words, assessments of thinking skills do a better job predicting which candidates will be successful in a given role than other forms of assessment.
The match between the complexity of national leaders’ thinking and the complexity level of the problems faced in their roles is important. While we will not be able to assess the actual complexity level of the thinking of national leaders, we will be able to examine the complexity of their responses to questions posed by prominent journalists. In upcoming articles, we’ll be sharing our findings and discussing their implications.
Arthur, W., Day, E. A., McNelly, T. A., & Edens, P. S. (2003). A meta‐analysis of the criterion‐related validity of assessment center dimensions. Personnel Psychology, 56(1), 125-153.
Becker, N., Höft, S., Holzenkamp, M., & Spinath, F. M. (2011). The Predictive Validity of Assessment Centers in German-Speaking Regions. Journal of Personnel Psychology, 10(2), 61-69.
Beehr, T. A., Ivanitskaya, L., Hansen, C. P., Erofeev, D., & Gudanowski, D. M. (2001). Evaluation of 360 degree feedback ratings: relationships with each other and with performance and selection predictors. Journal of Organizational Behavior, 22(7), 775-788.
Dawson, T. L., & Stein, Z. (2004). National Leadership Study results. Prepared for the U.S. Intelligence Community.
Dawson, T. L. (2017, October 20). Using technology to advance understanding: The calibration of CLAS, an electronic developmental scoring system. Proceedings from Annual Conference of the Northeastern Educational Research Association, Trumbull, CT.
Dawson, T. L., & Thornton, A. M. A. (2017, October 18). An examination of the relationship between argumentation quality and students’ growth trajectories. Proceedings from Annual Conference of the Northeastern Educational Research Association, Trumbull, CT.
Gaugler, B. B., Rosenthal, D. B., Thornton, G. C., & Bentson, C. (1987). Meta-analysis of assessment center validity. Journal of Applied Psychology, 72(3), 493-511.
Helbing, D. (2013). Globally networked risks and how to respond. Nature, 497, 51-59.
Hunter, J. E., & Hunter, R. F. (1984). The validity and utility of alternative predictors of job performance. Psychological Bulletin, 96, 72-98.
Hunter, J. E., Schmidt, F. L., & Judiesch, M. K. (1990). Individual differences in output variability as a function of job complexity. Journal of Applied Psychology, 75, 28-42.
Johnson, J. (2001). Toward a better understanding of the relationship between personality and individual job performance. In M. R. R. Barrick, Murray R. (Ed.), Personality and work: Reconsidering the role of personality in organizations (pp. 83-120).
McDaniel, M. A., Schmidt, F. L., & Hunter, J., E. (1988a). A Meta-analysis of the validity of training and experience ratings in personnel selection. Personnel Psychology, 41(2), 283-309.
McDaniel, M. A., Schmidt, F. L., & Hunter, J., E. (1988b). Job experience correlates of job performance. Journal of Applied Psychology, 73, 327-330.
McDaniel, M. A., Whetzel, D. L., Schmidt, F. L., & Maurer, S. D. (1994). Validity of employment interviews. Journal of Applied Psychology, 79, 599-616.
Rothstein, H. R., Schmidt, F. L., Erwin, F. W., Owens, W. A., & Sparks, C. P. (1990). Biographical data in employment selection: Can validities be made generalizable? Journal of Applied Psychology, 75, 175-184.
Stein, Z., Dawson, T., Van Rossum, Z., Hill, S., & Rothaizer, S. (2013, July). Virtuous cycles of learning: using formative, embedded, and diagnostic developmental assessments in a large-scale leadership program. Proceedings from ITC, Berkeley, CA.
Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Personality measures as predictors of job performance: A meta-analytic review. Personnel Psychology, 44, 703-742.
This morning, I received a newsletter from Sir Ken Robinson, a popular motivational speaker who focuses on education. There was a return email address, so I wrote to him. Here's what I wrote:
Dear Sir Ken,
"I love your message. I'm one of the worker bees who's trying to leverage the kind of changes you envision.
After 20+ years of hard work, my colleagues and I have reinvented educational assessment. No multiple choice. No high stakes. Our focus is on assessment for learning—supporting students in learning joyfully and deeply in a way that facilitates skills for learning, thinking, inquiring, relating and otherwise navigating a complex world. Our assessments are scalable and standardized, but they do not homogenize. They are grounded in a deep study of the many pathways through which students learn key skills and concepts. We're documenting, in exquisite (some would say insane) detail, how concepts and skills develop over time so we can gain insight into learners' knowledge networks. We don't ask about correctness. We ask about understanding and competence and how they develop over time. And we help teachers meet students "where they're at."
We've accumulated a strong base of evidence to support these claims. But now that we're ready to scale, we're running up against hostility toward all standardized assessment. It's difficult to get to the point where we can even have a conversation with our pedagogical allies. Ouch!
Lectica is organized as a nonprofit so we can guarantee that the underprivileged are served first. We plan to offer subscriptions to our assessments (learning tools) without charge to individual teachers everywhere.
We've kept our heads down as we've developed our methods and technology. Now we're scaling and want to be seen. We know we're part of the solution to today's educational crisis—perhaps a very big part of the solution. I'm hoping you'd like to learn more."
My email was returned with this message: "The email account that you tried to reach does not exist." How frustrating.
So, I thought I'd pen this post and ask my friends and colleagues to help me get access to Sir Ken's ear. If you know him, please forward this message. I'm certain he'll be interested in what we're doing for learning and development. Where are you Sir Ken Robinson? Can you hear me? Are you out there?
For several years now, one of our heroes, professor Howard Drossman of Colorado College and the Catamount Center, has been working with Lectical Assessments and helping us build LESA, the Lectical Environmental Stewardship Assessment.
Dr. Drossman's areas of expertise include developmental pedagogy, environmental stewardship, and the development of reflective judgment. His teaching focuses on building knowledge, skill, and passion through deep study, hands-on experience, and reflection.
For example, Dr. Drossman and ACM (Associated Colleges of the Midwest) offered a 10-day faculty seminar on interdisciplinary learning called Contested Spaces. This physically and intellectually challenging expeditionary learning experience provided participants with multiple disciplinary perspectives on current issues of land stewardship in the Pikes Peak region of Colorado.
A second, ongoing program is offered by Catamount Center and Colorado College is dedicated to inspiring the "next generation of ecological stewards." This program, called TREE (Teaching & Research in Environmental Education), is a 16-week, residential program for undergraduate students who have an interest in teaching and the environment. Program participants live and learn in community at the Catamount Mountain Campus, which is locatedin a montane forest outside of Woodland Park, Colorado. Through study and practice, they cultivate their own conceptions of environmental stewardship and respect for the natural world, while building skills for creating virtuous cycles of learning and useable knowledge in K-12 classrooms.
Dr. Drossman embeds Lectical Assessments in both of these programs, using them to customize instruction, support individual development, and measure program outcomes. He also is working closely with us on the development of the LESA, which is one of the first assessments we plan to bring online after our new platform, LecticaLive, has been completed.