From Augmented to AI Automated Evaluation
AI in Evaluation: From Augmented to Automated Evaluation
How artificial intelligence can support evaluation across the policy cycle—and the emerging possibility of integrating evaluation into autonomous AI systems.
Case study source: Steve Jacob, Artificial Intelligence and the Future of Evaluation: From Augmented to Automated Evaluation, Digital Government: Research and Practice.
Why AI matters for evaluation
Artificial intelligence is changing professional practices, including evaluation. The article examines how current AI applications can improve and enhance evaluation, how AI could affect the policy cycle, and how evaluation might eventually become part of autonomous AI systems that design, implement and evaluate public policies with minimal or no human supervision.
For evaluation, this creates a potential progression from AI-augmented evaluation, where AI supports human evaluators, toward more integrated feedback systems and, prospectively, automated evaluation.
AI-augmented evaluation
Generative AI can potentially be integrated into all stages of evaluation, from commissioning through presentation and dissemination.
At the planning stage, AI can help draft terms of reference, complete sections of templates, summarize program documentation and develop logic models. For evaluators, generative AI can support understanding of evaluative approaches, preparation of tools, empowerment evaluation activities, interview questions and illustrations. It can also help adapt findings, conclusions and recommendations for different audiences.
However, the article emphasizes that human expertise remains paramount, particularly for identifying and correcting errors and hallucinations in AI-generated outputs.
For another example of how generative AI can be used in evaluation practice, see the EvalCommunity case study on
Generative AI and Empowerment Evaluation.
AI across the evaluation lifecycle
The article describes potential applications of AI across several parts of the evaluation process.
Planning, scoping and design
- Background research
- Theories of change and logic models
- Indicator development
- Terms of reference
- Program summaries
Data collection and analysis
- Questionnaires and interview protocols
- Simulation
- Data collection
- Data cleaning and transformation
- Data analysis
- Drafting reports
Judgments and recommendations
- Assessing merit, worth and usefulness
- Developing suggestions
- Exploring scenarios
- Considering likely outcomes and risks
Reporting and dissemination
- Reports
- Videos and podcasts
- Translations
- Interactive visualizations and dashboards
AI for qualitative analysis
Natural language processing and generative AI can support the processing and analysis of qualitative data. The article cites an experiment involving CoLoop in which AI identified the same general themes in interview transcripts in approximately 10–12 seconds, compared with around a dozen hours for evaluators working manually.
The speed difference does not remove the need for evaluation expertise. The article notes that AI-generated outputs must be checked for accuracy.
Read the EvalCommunity case study on CoLoop qualitative research.
World Bank: AI-assisted analysis of project reports
The article also describes a World Bank application in which unsupervised machine learning was used to analyze 392 unique project reports from 64 countries receiving international aid. The analysis was used to identify factors associated with project success or failure.
According to the cited work, the approach identified coherent, domain-relevant and potentially novel topics and patterns that human analysis might have missed.
From evaluation to dynamic policy feedback
A traditional policy cycle can be represented as a sequence of problem identification, consideration of alternatives, adoption, implementation and evaluation. The article suggests that AI could make parts of this process more dynamic.
AI could support agenda-setting and policy scenario analysis, while faster data processing could allow evaluation findings to be incorporated more quickly into policy and program decisions.
The article introduces the concept of a Dynamic Public Policy Cycle (DPPC) in which AI could contribute to a more agile and incremental policy process. Importantly, the article states that the DPPC does not yet exist as such; it is presented as a future-oriented concept.
Kenya: AI and computer vision for COVID-19 monitoring
The article describes an application in Kenya in which machine learning and computer vision were used by hospitals to support COVID-19 management. The system measured factors including personal protective equipment and mask use, social distancing and gathering size, and generated a risk score.
The resulting data and risk score enabled program managers to adjust implementation in real time.
Read the EvalCommunity case study on AI and computer vision for COVID-19 monitoring in Kenya.
Gemma Services: real-time information for program management
Another example discussed in the article is Gemma Services in the United States. The organization explored the use of AI and analytics for social programs serving children and families experiencing severe emotional and behavioral challenges.
Working with a private consulting firm, the project developed causal, predictive, prescriptive and evaluative models. Practitioners received real-time information that could support the personalization of interventions, while managers received information to help consider how services could be combined for different subgroups.
The article presents this example as an illustration of how evaluation could move beyond an ex-post accountability function toward a more dynamic management tool.
Read the EvalCommunity case study on Precision Analytics, First Place for Youth and Gemma Services.
Toward automated evaluation
The article then considers a more distant possibility: integrating evaluation into autonomous AI systems.
Current systems discussed in the article still require human input. Autonomous AI systems capable of operating with minimal human supervision are described as being in their infancy.
In the article’s illustrative scenario, an autonomous AI system could implement a public subsidy program, document grants and outcomes, collect and analyze data, identify subgroups, vary incentives or penalties, continuously assess outcomes and make corrections.
Evaluation results could also be incorporated directly into the data used by such systems, potentially creating rapid feedback loops between implementation and evaluation.
The article presents autonomous public-policy systems and automated evaluation as future possibilities. It does not describe fully autonomous evaluation as an established practice.
AuroraAI in Finland
The article uses AuroraAI in Finland as an early-stage example of a possible direction for intelligent public-service ecosystems. Fully autonomous systems do not currently exist, according to the article.
AuroraAI is described as a network of intelligent services connecting public, private and third-sector services and offering dynamic service chains based on real-time data.
Evaluation is an important component of the approach. The article notes that data generated within such a service ecosystem could potentially improve understanding of real-world impacts and help steer operators.
Risks and requirements
The potential benefits of AI in evaluation come with technical, ethical and legal risks. The article emphasizes that the evidence base remains limited and that further experimentation is needed.
Human factors
Evaluators need appropriate knowledge, capacity and direct experience with AI, as well as the ability to use AI securely. Risks include security breaches, algorithmic bias and problems involving sensitive data and privacy.
Data and technical quality
AI applications depend on quality data, appropriate context and expert supervision. The article also highlights the challenges of black-box systems and limited transparency or explainability. When accuracy cannot be verified or outputs cannot be adequately explained, evaluation rigor and transparency may be compromised.
Bias and overconfidence
Training data can contain biases that affect AI outputs. The article therefore emphasizes the importance of quality and representative data, as well as training people to understand AI limitations and work effectively with AI systems.
What this means for M&E professionals
The article suggests that AI could eventually support evaluation throughout the policy and program cycle rather than being used only for isolated analytical tasks.
- AI can support evaluators: Human expertise remains important for judgment, quality assurance and identifying errors or hallucinations.
- Evaluation may become more continuous: Faster data processing could support feedback during implementation rather than only after a program has ended.
- Evidence could become more integrated: AI-enabled systems could potentially connect implementation data, analysis and evaluation findings more closely.
- Responsible use remains essential: Privacy, confidentiality, transparency, explainability, data quality and staff capacity are central considerations.
- Automation is still prospective: The article describes fully autonomous evaluation as a future possibility rather than a current standard practice.
A practical checklist for AI-enabled evaluation
- Define where AI is being used within the evaluation process.
- Maintain human oversight of AI-generated outputs.
- Check AI outputs for accuracy, errors and hallucinations.
- Consider privacy, confidentiality and data security.
- Assess the quality and representativeness of the data being used.
- Consider whether the AI system is sufficiently transparent and explainable for the intended use.
- Ensure evaluators and other staff have appropriate AI knowledge and capacity.
- Distinguish current AI applications from future or experimental possibilities.
The bigger picture
The article concludes that current AI use in evaluation remains limited to a relatively small number of experimental studies. At the same time, AI’s capabilities in data processing and simulation could support more continuous evaluation across the policy and program cycle. Realizing this potential requires responsible and transparent use of AI, attention to confidentiality and privacy, and appropriate training for public-sector staff.
Source
Jacob, S. Artificial Intelligence and the Future of Evaluation: From Augmented to Automated Evaluation. Digital Government: Research and Practice.
View the original article on ACM.
Source note: This case study distinguishes documented applications from forward-looking concepts discussed in the source article. Examples involving autonomous evaluation are presented as prospective possibilities, not established practices.
