How NRC Built a Chatbot to Transform Access to M&E Knowledge
AI-Powered Knowledge Retrieval: How NRC Built a Chatbot to Transform Access to M&E Knowledge
The Norwegian Refugee Council (NRC) has developed a prototype AI-powered chatbot to address a persistent challenge in humanitarian operations: efficiently accessing policies, evaluation frameworks, and program guidance from extensive internal databases. Using a small, cost-effective Llama-based large language model hosted in NRC's own cloud environment, the chatbot provides instant, accurate answers to staff queries — dramatically reducing time spent searching for information and enabling faster, more evidence-based decision-making across monitoring, evaluation, and program implementation.
Context: The M&E Knowledge Gap in Humanitarian Organizations
The Norwegian Refugee Council (NRC) operates in some of the world's most complex humanitarian environments. Staff members constantly need rapid access to policies, guidelines, evaluation frameworks, and program documentation. However, effectively preserving and accessing institutional knowledge is a persistent challenge for non-profits, given restricted resources, frequent organizational changes, and high staff turnover.
The core challenge identified by NRC's AI Lead, Zineb Bhaby:
"Users working in programs often ask about ways to easily retrieve information they need for their work. For example, they might be looking for a specific policy or a document with approaches on provision of cash assistance."
This knowledge retrieval problem has direct consequences for Monitoring and Evaluation. When staff cannot quickly find evaluation frameworks, monitoring tools, or guidance documents, the result is inconsistent data collection, delays in evidence-based decision-making, and the risk of using outdated or incorrect methodologies.
The M&E Problem: When Knowledge Is Hard to Access
From a Monitoring and Evaluation perspective, NRC faced several interconnected challenges that affected the quality and consistency of its work:
Staff spent excessive time searching for methodologies, indicators, and reporting templates.
Organizational knowledge was scattered across large internal databases with no intuitive search interface.
Slow information retrieval meant slower responses in humanitarian contexts where timing is critical.
Without version control and easy access to current policies, teams risked working from obsolete documents.
These challenges directly affect monitoring quality, evaluation consistency, organizational learning, and the speed of evidence-informed decision-making — all core functions of a robust M&E system.
The AI Solution: An LLM-Powered Knowledge Retrieval Chatbot
NRC believed that artificial intelligence — specifically large language models (LLMs) combined with a chatbot user interface — could address the knowledge retrieval problem. The team conducted a thorough assessment of technical capabilities and data infrastructure to ensure readiness for AI implementation, mapping potential use cases identified by both internal stakeholders and other humanitarian organizations. Knowledge management and retrieval emerged as the priority due to its potential to support program operations and save users' time.
How the system works
The chatbot is built using a Llama-based model (a small, efficient LLM) hosted entirely in NRC's own cloud environment. It is trained exclusively on NRC's internal documents — policies, program guidance, evaluation reports, and technical documents — ensuring that responses are based only on verified organizational knowledge. Users interact through a natural language chatbot interface, asking questions and receiving instant answers with links to source documents for validation.
Reduces computing costs and environmental impact
Ensures data security and avoids vendor lock-in
Small enough to potentially run on local servers
Prioritizes accurate information extraction over content creation
M&E Integration: How the Chatbot Supports Monitoring, Evaluation, and Learning
The NRC chatbot is not just an IT tool — it is an M&E enabler. By accelerating access to knowledge, it strengthens every component of the M&E system:
Monitoring Support
Staff can instantly access monitoring tools, indicator guidance, and data collection procedures — improving consistency and quality of field-level data collection. Instead of searching for hours, they retrieve the correct methodology in seconds.
Evaluation Support
Evaluators can quickly retrieve evaluation methodologies, past evaluation reports, and organizational standards. This reduces errors, saves time, and ensures that evaluations are grounded in the organization's established frameworks and lessons from previous work.
Learning and Knowledge Management
The chatbot centralizes access to institutional knowledge, making it easier for new staff to learn and for experienced staff to share. With continuous updates through data curation and user feedback, the system helps create a true learning organization.
Decision-Making Support
Field teams and program managers can access guidance instantly, applying correct methodologies and evidence-based approaches in real time. This accelerates humanitarian response and improves the quality of decisions.
Technical Design: Building for Security, Cost, and Connectivity
NRC made several strategic technical choices to ensure the solution would be practical for humanitarian contexts:
- Small LLM (Llama-based): The team chose a small language model because the use case focused on accurately retrieving information from internal documents, not generating new content. A small LLM was powerful enough for the task and significantly more cost-effective.
- Self-hosted in NRC's cloud environment: This keeps NRC's data secure, avoids vendor lock-in, and gives the organization full control over its AI infrastructure.
- Potential for local server deployment: The solution is small enough to potentially run on a local server in low-connectivity environments — critical for field operations in remote humanitarian settings.
- Trained exclusively on internal NRC documents: The model was solely trained on NRC's internal information to avoid introducing any inaccurate information from external sources.
The team built a working prototype within three months, which was then opened to a select group of users for testing.
Results and Early Outcomes
Key M&E Impact Areas
| Area | Improvement |
|---|---|
| Monitoring | Faster access to tools and indicators → more consistent data collection |
| Evaluation | Quick retrieval of methodologies → better consistency across evaluations |
| Learning | Centralized knowledge access → stronger organizational learning |
| Decision-making | Instant access to guidance → faster, evidence-based decisions |
Challenges and Risks (Critical for Credibility)
The NRC case study honestly acknowledges several important limitations and risks of the AI-powered approach:
Need for expert validation
The project team learned that the tool really needs knowledge owners — people very familiar with specific policies and processes — to validate the output. This takes time and needs to be factored into building such an application.
Risk of incorrect information merging
Without expert review, the tool may combine information from different documents on different subject matters into one response that is not quite accurate. This is why responses always include links to source documents.
Data curation complexity
The team developed a data curation strategy defining which documents would be ingested and how to handle different versions of the same policy. Clean datasets and version control are essential.
Scaling requires additional resources
While the prototype has low running costs, scaling to more users and adding use cases will increase infrastructure costs. Additional funding will be needed for broader organizational rollout.
Key Lessons for M&E Professionals
AI accelerates access to M&E knowledge
The NRC chatbot demonstrates that AI does not replace M&E expertise — it accelerates access to the knowledge that enables better monitoring, evaluation, and learning.
Retrieval-focused AI is safer than generative
By designing the system to retrieve existing information rather than generate new content, NRC reduced the risk of hallucination and maintained control over the accuracy of responses.
Expert validation is non-negotiable
The NRC team learned that knowledge owners must validate AI outputs. Without this, there is a real risk of the tool merging unrelated information or referencing outdated documents.
Small LLMs can be sufficient and cost-effective
Not every AI solution requires a massive model. For information retrieval from a defined knowledge base, a small LLM hosted in your own environment can be powerful, secure, and affordable.
From searching to instant retrieval — from fragmented knowledge to centralized intelligence
The NRC case shows that AI in M&E is not only about prediction and advanced analytics. It is also about access, knowledge, and decision support. By transforming how staff access institutional knowledge, the chatbot enables faster, more consistent, and more evidence-based M&E practice across the organization.
Future Developments
NRC has ambitious plans for expanding and improving the chatbot:
- Measuring time saved: The team wants to assess actual "usefulness" through measures such as time saved by using the tool versus manually searching for relevant information.
- Expanding use cases: The chatbot could support grant writing, donor reporting, and other knowledge-intensive tasks.
- Adding external sources: The team has tested interfacing with ReliefWeb (run by UN OCHA), whose content is carefully curated and considered reliable.
- Sharing with partners: A future vision is to share the chatbot with other organizations that NRC collaborates with.
- Continuous monitoring of AI developments: The team will continue to assess newer LLMs that may be more suitable for this or other use cases.
Frequently Asked Questions
How is this chatbot different from public LLMs like ChatGPT?
Unlike public LLMs trained on general internet data, NRC's chatbot is trained exclusively on NRC's internal documents. It is designed to retrieve existing information, not generate new content, and includes strict guardrails against hallucination. It is also self-hosted in NRC's cloud, ensuring data security and privacy.
How does the tool prevent hallucinations?
The team implemented strict guardrails and system prompting to ensure the model does not generate content outside the scope of the knowledge base. Responses always include links to source documents so users can double-check and validate responses.
Can this approach work in low-connectivity environments?
Yes — the solution was designed to be small enough to potentially run on a local server in low-connectivity environments, making it suitable for field operations in remote humanitarian settings.
How can I learn more about NRC's AI work?
Contact Zineb Bhaby, NRC AI Lead, at Zineb.bhaby@nrc.no. The case study was written by Meheret Takele Mandefro, Business Analyst at NetHope Center for the Digital Nonprofit, and was funded by the UK Humanitarian Innovation Hub.
AI as an M&E Enabler, Not Replacement
The NRC knowledge retrieval chatbot represents a practical, responsible approach to AI in humanitarian M&E. Rather than aiming for flashy predictive analytics, the team focused on a core operational challenge — accessing internal knowledge — and built a solution that is secure, cost-effective, and designed for real-world humanitarian constraints.
The key insight for M&E professionals is clear: AI does not replace M&E expertise — it accelerates access to the knowledge that enables better monitoring, evaluation, and learning.
As the NRC team continues to develop and scale this tool, they offer a model for other humanitarian organizations seeking to leverage AI for knowledge management and decision support — always with human expertise at the centre.
