One of the biggest criticisms of organizational data cultures is that organizations have plenty of data, it’s just that they don’t use it. And, if you look at the evidence, those critics might be right. If no policy changed, the program wasn't refined, nothing new launched, and budgets weren’t adjusted, it’s easy to conclude the available data wasn’t consulted.
This pattern of measuring assessment by immediate action alone shows up across sectors. For years, higher education has measured assessment success by looking for immediate, observable actions. Did the department change the curriculum? Did leadership approve a new initiative? Did the assessment "close the loop?" Did our constituents learn more? Corporate organizations aren't so different. For years, they have measured assessment success by looking for immediate, observable actions. Did customer feedback lead to a product change? Did leadership approve a new go-to-market strategy? Did the research "close the loop?" Did our customers get more value?
Indeed, it makes sense for organizations to focus on these end goals, but they aren’t the only ways that data can inform and impact organizational change. Some argue that focusing only on immediate actions dramatically underestimates the many possible impacts of assessment. Beyond supporting decision-making, data shapes conversations, changes understanding, and builds or breaks down partnerships. In other words, assessment can transform how people think.
Before you dismiss an analysis, dashboard, or data-related project as unsuccessful because it didn't produce an immediate decision, consider a different question: How can this assessment influence the people who engage with it?
Four Ways to Use Data to Influence the Organization
Assessment can influence organizations beyond immediate action. Rather than evaluating every assessment by whether it produced a specific change, consider these different types of influence (Jonson et al., 2014; Kirkhart, 2000).
Instrumental influence is the kind of data use most people expect from projects. This kind of use leads to direct changes. Examples of instrumental influence include revising a curriculum, changing a training requirement, implementing a new intervention, or adjusting policy.
Sometimes, assessment changes understanding before it changes behavior. Conceptual influence is the kind of data use where new ways of thinking are being generated. Perhaps decision-makers still aren’t ready to act, but they’re warming up to the ideas. Changes in understanding of a problem can be just as important - if not more important - than taking a specific action.
Affective influence refers to changing attitudes around assessment processes themselves. For example, consider staff completing a staff engagement survey. If organizational leaders honor their anonymity and follow through on ideas presented in the engagement, then staff are likely to feel more positively towards staff engagement survey opportunities. If organizational leaders act defensively, try to match anonymous responses with individual staff members, or other unproductive actions, staff disposition towards participating in survey opportunities may turn negative. Positive affect towards data processes and evidence can be highly useful to [something].
Finally, many assessment processes ultimately involve the act of confirming you’re on the right track. Affirmative influence focuses on confirming the effectiveness or appropriateness of the current path. Sometimes the data provides confidence that current strategies are effective and worth continuing.
With these influences in mind, think back to one of the recent projects in your organization:
Did stakeholders develop a deeper understanding of the issue?
Did new partnerships emerge?
Did conversations shift from opinions to evidence?
Did the process improve how future evidence will be collected?
Did leaders become more confident in the data?
Did stakeholders feel more bought into the process?
These things may not emerge in a documented action plan, but they may be the reason future reason is even possible.
When we consider all four of these influences, we also get a better sense of the possible impacts and results of using data. Not only can data be used as part of improving student, staff, or organizational learning, but it can also lead to personal transformations of stakeholder beliefs, it can build or strengthen formal or informal communities of practice, and when results are used to generate or sustain support for specific initiatives, policies, or decisions, data can also be symbolic or political. Importantly, these influences may not be immediately obvious, which is why it is important to consider data use as unfolding over time.
Keeping Data Use at the Forefront of Assessment Practice
Recognizing these different forms of influence is one thing; designing assessment work to encourage them is another. As you prepare your next evaluation project, consider a few ways to expand how you and your collaborators think about data use.
Manifest influence before you begin. When planning an assessment, ask what you hope will be different because people engaged with the evidence. A new decision or program change might be the goal, but perhaps you also want stakeholders to better understand an issue, feel more confident in the evidence, or confirm that an existing strategy is working. Naming these possibilities early makes it easier to recognize them later.
Create opportunities for people to engage with the evidence. Data is unlikely to influence an organization simply because a report was distributed or a dashboard was published. Build opportunities for stakeholders to discuss findings and connect the evidence to their own work. Sometimes the conversation around the findings produces as much data use potential as the findings themselves.
Look for influence throughout the process, not just at the end. The process of assessment (especially collaborative assessment) can reveal meaningful data use opportunities that may otherwise be missed. Pay attention to what happens while discussions about the data are underway. Are stakeholders asking different questions? Are people from different areas beginning to work together? Has skepticism about the data decreased? These shifts can signal conceptual or affective influence well before an instrumental decision occurs.
Document more than decisions. When recording the outcomes of an assessment, don’t stop once you’ve reported the ultimate outcome. Capture the conversations that emerged, the questions stakeholders began asking, and the partnerships that developed throughout the process. These indicators may seem less tangible than a policy change or curriculum revision, but they often reveal whether your assessment is truly influencing the organization. A stronger narrative of data use might capture changes in understanding, relationships, confidence, and conversation alongside those concrete, instrumental actions.
Ultimately, if we only use immediate action as the benchmark of a “data use success”, we’re missing some of the biggest possible contributions data has to influence our organization: the shifts in understanding, attitudes, relationships, and confidence that eventually make actions and improvements possible.
As you prepare your next assessment report, consider documenting more than just the decisions that resulted from your work. If we broaden our definition of success, we just might gain a more complete picture of how evidence contributes to continuous improvement. So, the next time someone says, “We have plenty of data; we just don’t use it,” consider whether the data truly went unused or whether its influence is still waiting in the wings, waiting to be seen.
References
Jonson, J. L., Guetterman, T., & Thompson Jr, R. J. (2014). An Integrated Model of Influence: Use of Assessment Data in Higher Education. Research & Practice in Assessment, 9, 18-30.
Kirkhart, K. (2000). Reconceptualizing evaluation use: An integrated theory of influence. New Directions for Evaluation, 88(1), 5–24.
