EvalCommunity Tutorial

How GenAI Works: What Evaluators Need to Know Before Using AI

A practical explanation of how generative AI and large language models work, why they can hallucinate, and what evaluators should check before using AI in monitoring, evaluation, and research.

Tutorial Summary

Generative AI can support evaluation work by drafting, summarizing, translating, organizing, coding, and exploring information. But it does not work like a search engine, researcher, or evaluator.

This tutorial explains how GenAI works, why outputs can be inaccurate or biased, and how evaluators can use AI responsibly while protecting evidence quality, accountability, inclusion, and transparency.

What You Will Learn

What generative AI and large language models are.
How GenAI generates text through next-token prediction.
Why GenAI is a statistical system, not a knowledge authority.
Why AI tools can hallucinate or invent unsupported claims.
Why bias, dominant narratives, and representation matter in evaluation.
Why source attribution and evidence traceability are critical.
How RAG can improve grounding but does not eliminate verification risks.
Why explainability, copyright, and replicability are important for evaluation practice.
How evaluators can apply responsible safeguards when using GenAI.

Authoritative Sources Used

This tutorial is based on selected research and official guidance on large language models, hallucination risks, retrieval-augmented generation, bias in qualitative analysis, and copyright issues related to generative AI.

How GenAI Generates Text

This simplified workflow shows how a large language model turns a prompt into a generated response.

Step 1

Prompt

The user gives an instruction, question, or document.

Step 2

Context

The model uses the prompt and available context.

Step 3

Prediction

The model predicts likely next tokens.

Step 4

Generation

The response is built token by token.

Step 5

Verification

A human checks evidence, bias, and accuracy.

1. Why Evaluators Need to Understand How GenAI Works

Generative AI is becoming part of evaluation practice. Evaluators may use it for summarizing reports, translating text, drafting tools, coding qualitative data, identifying themes, reviewing evidence, or preparing learning products.

However, many risks come from misunderstanding what these systems actually do. A large language model does not know a topic in the same way a researcher or evaluator does. It generates text by predicting likely patterns in language.

Key Principle for Evaluators

Use GenAI as a support tool, not as a source of truth. Evaluation evidence should still come from documents, datasets, interviews, observations, monitoring data, verified sources, and human analysis.

2. What Is GenAI?

Generative AI refers to AI systems that can create new content, including text, images, audio, video, code, summaries, and structured outputs. In evaluation practice, the most common form is text-based GenAI powered by large language models.

GenAI Can Support

Drafting text
Summarizing documents
Translating content
Classifying responses
Suggesting themes
Generating code or tables
Preparing report outlines

“`

GenAI Does Not Automatically

Verify facts
Assess source credibility
Understand local context
Protect confidentiality
Represent marginalized voices fairly
Make evaluation judgments
Guarantee replicable outputs

“`

3. What Are Large Language Models?

Large language models, or LLMs, are AI models trained on very large collections of text. They learn statistical patterns in language, including grammar, syntax, style, word relationships, and common associations between concepts.

Training data may include web data, books, code repositories, news archives, scientific papers, encyclopedia-style content, forums, or other licensed, public, scraped, or user-provided data depending on the model and provider.

Important Caution

Not all AI providers disclose full training data sources. Evaluators should not assume that a model’s training data is complete, balanced, ethical, representative, or suitable for a specific evaluation context.

4. GenAI Is a Statistical System, Not a Knowledge Authority

GenAI models generate plausible language based on statistical patterns. They do not evaluate evidence like a researcher, compare source credibility like a literature reviewer, or interpret social meaning like an evaluator.

System or Role	What It Does	Evaluation Implication
Search engine	Retrieves sources or web pages.	Sources still need credibility review.
Researcher	Compares, evaluates, and interprets evidence.	Human expertise remains necessary.
Evaluator	Interprets evidence in context and makes judgments.	Accountability cannot be delegated to AI.
GenAI model	Generates likely language based on patterns.	Outputs require verification before use.

5. Temperature and Output Variation

Temperature is a setting that influences how predictable or varied an AI output may be. A lower temperature usually produces more conservative and consistent responses. A higher temperature can produce more varied, creative, or exploratory responses.

Lower Temperature

More predictable
More conservative
Better for structured outputs
Useful for consistency checks

“`

Higher Temperature

More varied
More creative
Useful for brainstorming
Riskier for analysis or reporting

“`

For evaluation tasks, more controlled outputs are usually safer when accuracy, coding consistency, traceability, or reporting quality matter.

6. Why GenAI Can Hallucinate

A hallucination occurs when an AI system produces information that is false, unsupported, fabricated, or misleading while presenting it confidently.

Common Hallucination Risks in Evaluation

Fabricated references or citations.
Unsupported evaluation findings.
Incorrect summaries of reports or transcripts.
False statistics or invented figures.
Overconfident recommendations.
Misinterpretation of qualitative evidence.
Plausible-sounding but inaccurate conclusions.

Evaluation Safeguard

Every AI-generated claim used in an evaluation product should be checked against a source document, dataset, interview quotation, coded segment, monitoring record, or verified reference.

7. Attribution and Source Awareness

Many GenAI tools generate answers without showing where each statement came from. This is a serious challenge for evaluation because findings must be traceable to evidence.

An AI-generated paragraph may sound like synthesis, but without attribution it is difficult to judge credibility, accuracy, relevance, or bias.

Traceability Rule

Do not use AI-generated analysis as evidence. Use the original document, dataset, interview quote, coded segment, field note, or verified source as the evidence.

8. What RAG Changes and What It Does Not Solve

Retrieval-Augmented Generation, or RAG, connects a language model to external documents, databases, websites, or knowledge bases. Instead of relying only on model training, the system retrieves relevant information and uses it to generate an answer.

RAG Can Improve

Source grounding
Citation support
Access to recent information
Use of organizational documents
Evidence traceability

“`

RAG Does Not Eliminate

Poor source quality
Misinterpretation of text
Missing context
Biased source collections
Incorrect citation use
Overconfident synthesis

“`

For evaluators, RAG is most useful when it connects AI outputs to verified reports, interview transcripts, monitoring data, policies, or evidence repositories. However, retrieved sources still need human review.

9. Bias and Dominant Narratives

Large language models learn from existing texts. If those texts overrepresent dominant languages, institutions, regions, or perspectives, the model may reproduce those patterns in its outputs.

Risks for Evaluation

Reinforcing stereotypes.
Overlooking marginalized voices.
Treating dominant narratives as neutral.
Missing local context.
Producing generic recommendations.
Misinterpreting culturally specific language.
Underrepresenting minority perspectives.

Evaluation often aims to surface diversity, lived experience, and power dynamics. AI can weaken this if it smooths out disagreement, removes nuance, or privileges the most statistically common patterns.

10. Explainability and the Black Box Problem

Large language models operate through complex neural networks with many parameters. Their outputs emerge from statistical relationships that are difficult to trace as step-by-step reasoning.

This creates a black box problem. We can see the prompt and the output, but we cannot fully trace why the model produced that exact output.

Evaluation Implication

If an AI tool suggests a theme, code, or recommendation, the evaluator must justify it using source evidence, not by referring to the AI’s reasoning.

11. Replicability and Consistency Problems

Unlike traditional rule-based tools, GenAI does not always produce the same answer for the same prompt. Outputs can vary because these systems are probabilistic.

Why Outputs Can Vary

Temperature settings.
Prompt wording.
Model updates.
System instructions.
User settings.
Interaction history.
Uploaded context.
Tool configuration.

Replicability Safeguards

Save prompts.
Save model or tool name and version where available.
Save outputs.
Use structured templates.
Run small sample checks.
Use human coding review.
Document human corrections.
Do not rely on AI alone for final findings.

12. Copyright, Consent, and Training Data Ethics

Many GenAI systems were trained on large collections of text, images, code, and other content. Some of this content may come from public web sources, books, code repositories, social media, licensed datasets, or other sources. This raises legal and ethical questions.

Questions Evaluators Should Consider

Was the training content used with permission?
Were authors or communities aware their content could be used?
Were copyrighted works included?
Were social media posts used without meaningful consent?
Are marginalized communities represented fairly?
Could AI outputs reproduce protected styles, structures, or language without attribution?
Are legal standards still evolving?

Evaluators should be especially cautious when using GenAI outputs in published reports, public learning products, course materials, or donor-facing documents.

13. Practical Implications for Evaluation Practice

AI Can Support	AI Should Not Be Used Alone For
Drafting and brainstorming	Final findings
Summarization and translation	Sensitive data interpretation
Coding support and thematic exploration	Attribution of causality
Report structuring and evidence organization	Ethical judgments
Learning product preparation	Final recommendations without human validation

14. Responsible Use Checklist for Evaluators

Is the output traceable to evidence?
Did the AI invent any facts, citations, or references?
Were sources checked?
Was sensitive data protected?
Could the output reinforce bias?
Are marginalized voices represented?
Were contradictions included?
Is the output too generic?
Can another evaluator review the same evidence?
Was the prompt and output saved?
Was AI use disclosed where appropriate?
Was final interpretation done by a human evaluator?

15. Practical Exercise for Learners

Exercise: Testing the Limits of GenAI in Evaluation

Use one short evaluation question, one project description, five stakeholder comments, and one short report excerpt.

Ask AI to summarize the report excerpt.
Compare the summary with the original text.
Identify omissions or overstatements.
Ask AI to identify themes in the stakeholder comments.
Check whether minority views are preserved.
Ask AI to generate one finding.
Check whether the finding is supported by evidence.
Ask AI for sources on a topic.
Verify whether the sources are real.
Write a reflection on what AI did well and what required human correction.

16. AI Use Statement for Evaluation Reports

GenAI tools were used to support selected drafting, summarization, or analytical support tasks. All AI-generated outputs were reviewed, edited, and validated by human evaluators. Final findings, interpretations, conclusions, and recommendations were based on source evidence and professional evaluation judgment. AI-generated outputs were not treated as standalone evidence.

17. Frequently Asked Questions

Does GenAI understand what it writes?

Not in the same way humans understand. GenAI generates text based on statistical patterns. It can produce coherent language, but it does not evaluate meaning, evidence, or truth like a human researcher or evaluator.

Why does GenAI hallucinate?

GenAI can hallucinate because it is designed to generate plausible language, not verify facts. When information is missing or uncertain, it may still produce a confident answer.

Is GenAI the same as a search engine?

No. A search engine retrieves sources. GenAI generates text. Some AI tools can use search or retrieval systems, but generated outputs still need verification.

What is RAG?

RAG stands for Retrieval-Augmented Generation. It connects an AI model to external documents, databases, websites, or knowledge bases so the model can generate answers using retrieved information.

Does RAG eliminate hallucinations?

No. RAG can improve source grounding, but it does not eliminate errors, misinterpretation, weak sources, missing context, or overconfident synthesis.

Can evaluators use GenAI for qualitative analysis?

Yes, but carefully. GenAI can support coding, summarization, and theme exploration, but final analysis should be reviewed and validated by human evaluators.

18. Final Quality Checklist

The article explains that GenAI predicts likely language rather than verifying truth.
The distinction between AI output and evaluation evidence is clear.
Hallucination, bias, attribution, explainability, copyright, and replicability risks are covered.
RAG is explained without overstating its benefits.
Evaluation safeguards are included throughout the tutorial.
AI use is framed as support for evaluators, not replacement of evaluators.
The FAQ answers common learner questions directly.
The AI use statement can be adapted for evaluation reports.

Conclusion

Generative AI can be useful in evaluation practice, but it is important to understand how it works. Large language models generate language by identifying statistical patterns and predicting likely sequences of text.

The responsible use of GenAI in evaluation is not about replacing evaluators. It is about helping evaluators work more efficiently while keeping human judgment, source verification, ethical reasoning, and contextual understanding at the center.