Learning from the Elders: Key points for navigating CVE evaluations
I’m not proud of the fact that my professional perspective has been shaped by eight years of working in war zones. But that’s what happened.
While my experiences monitoring NGO and community development in Iraq were most often conditions of outright war, my first exposure to the insurgency in Afghanistan was in the form of a kindly old man from a village outside the western city of Herat.
At a small social engagement hosted by the provincial head of the Ministry of Justice, the old man remained largely silent but listened attentively to our dinner conversations. As we pondered how linking Afghanistan’s formal justice system with its local traditions of restorative justice could help build an effective and legitimate state, I recall him often putting his hand to a silvery beard that extended several inches below his chin.
He approached during our goodbyes and explained through interpretation that he was a schoolteacher in his village, in which there were often many Taliban.
I have observed you carefully, and you seem like an honorable and sincere man, he said. Come to my village and I will introduce you to the Taliban to make peace.
Startled, I turned to my host and asked if it was okay for me to visit the schoolteacher’s village.
Certainly, my host said. As long as you don’t plan to return.
The encounter comes to mind whenever I am asked to support U.S. Government efforts to measure, analyze, and understand the vulnerabilities and resiliencies to violent extremism. The case of the kindly old man who may or may not have identified with the Taliban encapsulates some of the well-known challenges around analyzing violent extremism:
- Religious conservatism or even fundamentalism is not extremism
- Extremist belief can be deeply idiosyncratic, a product of shared consensus, tied to geo-political conditions, or all three
- Individual factors further interact across a varied landscape of community and societal dynamics which easily shift based on everyday events
It has been a privilege working with a few of the dedicated officers of the U.S. State Department’s Bureau for Conflict and Stabilization Operations (CSO) who are seeking to understand the VE phenomenon across a variety of contexts, each with their own complex dynamics.
While I have several times gotten lost in the proverbial fog of war when wrestling with demons in the data, a few broad points of emphasis have become clear:
Look at your data
Don’t bury your audience under tables, but actually look at your data. With the right visualization, the eye will be able to more effectively process contrasting information when it’s presented with color, space and design.
Use index variables, but only under the right conditions
When it comes to data, there are always too many variables to measure, and some are too similar to one another. Bundle them into an index to ease issues related to multiple comparisons, stabilize the variance, and increase confidence that any variation in the composite variable is a genuine signal, not random noise. If there’s movement, unpack the index to see if certain items are driving that change.
Use multiple regression to test the relationship of explanatory factors to a VE outcome
Look at your survey frequencies, alone and against several disaggregations. But to assess a durable relationship between an explanatory measure and an outcome, make sure you include other available information. This shifts your question from ‘What is the relationship between X and Y?’ to ‘What is the relationship between X and Y, after already knowing about Z?’ This reassures the researcher that the relationship of interest isn’t polluted by other relationships.
Rank variable importance to help identify the ‘true’ model
As more variables are examined simultaneously in multiple regression, they may start to have the effect of describing the data without actually describing the ‘true’ relationships that are hypothesized to exist in the real world. This has the practical effect of reducing the applicability of the analytical model to new data, because it fits too tightly to the idiosyncrasies of your current data.
Resampling-based methods known as model averaging or random forests crawl through your data and rank which variables contribute the most to predicting the outcome of interest. This ranking helps the researcher distinguish between what analytical model works best across datasets, and which works best to describe only the data at hand.
Ranking variable importance also helps distinguish between variables that may have a stronger relationship with the outcome in your data but do not necessarily predict the outcome in new data, and vice versa. As an example, the quality of public services has a weak relationship to VE sympathy, but model averaging suggests that the relationship is more fundamental to explaining VE sympathy than a multiple regression model would have you believe.
Imagine models that govern what’s happening. Many of them. Then hypothesize, test, and hypothesize again.
Multiple regression runs into the kitchen sink problem of everything going into the model and the model output becoming indecipherable. Prior theory to guide the choice, handling, and interpretation of variables is essential. Forgive the violence your simple model does to reality, but use its simplicity to explore bits and pieces of the reality you’re trying to describe. The blunt instruments of our analytical tools will not be able to fit the many pieces together, but a community of researchers might.
Explore complex interactions and the existence of sub-groups
Multiple regression can be great for identifying simple and broad relationships among a few variables. It’s lousy at letting us know how several variables may interact to define interesting sub-groups.
Routines known as regression or classification trees explore the intricacies of your data with the express purpose of picking out pathways of item responses that lead to wide variation in the outcome measure of interest. Understanding this variation has been eye-opening for spotting small groups of respondents with particularly strong vulnerabilities or resiliencies to VE.
Ask delicate questions delicately
Radicalized beliefs are deeply personal and by definition illicit – why would a respondent share such a belief with a stranger? Answering could also be dangerous if tied to a violent actor who wields some influence in the respondent’s community.
To ask indirectly, try linking a set of violent actors to a hypothetical policy or value statement. Then examine the variation in support for these propositions by each actor, and interpret the differences as latent measures of support. Such measures don’t give an outright statement of support for a VE actor, but they do offer relative measures of support and mitigate sensitivity and safety concerns involved with direct questions.
What we’ve found
MSI, on behalf of the Conflict and Stabilizations Operations Bureau, is midway through a set of surveys and focus groups looking at VE drivers in Bosnia Herzegovina, Philippines, Malaysia, Kenya, Somalia, and Niger. A lot of what we learned reassuringly echoes what we thought we already knew, and some learning has been new and surprising.
Findings to date confirm the absence of a link between poverty and VE sympathy, as well as the importance of distinguishing between religious intolerance and religious devotion among respondents who identify strongly with their faith.
More surprising has been the roles of sociality, a sense of personal worth, and one’s outlook on other people in helping to distinguish a radical belief from a merely conservative belief. A cause for concern and further inquiry is suggestive evidence of a potential role for government institutions or elected offices in tacitly endorsing and inflaming radical belief.
Understanding violent extremism is an ongoing conversation.
On reflective moments, I imagine scenarios where I visit the schoolteacher’s village and do return.
Who would I meet? What would they tell me, and what would I tell them? Which common values would we share, and where would we occupy parallel and irreconcilable universes? Would both parties see themselves as speaking truth to power, and could rapprochement be possible after such a violent collision between two truths and two powers?
One could say my journey toward understanding violent extremist belief and action is really just the imaginary conversations I have with the Taliban schoolteacher who says he’s ready to make peace.
Let’s keep talking.
Blog posts on the MSI website represent the views of the authors and do not necessarily represent the views of MSI.