A Lesson from Radiology: Task-Based Evaluation over Quizzes
The clearest template in the corpus for evaluating an AI-literacy program is a within-subjects experiment in radiology: no-AI vs on-demand AI vs integrated AI on a real diagnostic task (8 residents, 150 CXRs). It measured both upskilling and deskilling risks.
3D: Diagnostische Aufgabe (Radiologie-Template)
1. The experiment (Guide 2, §3)
- Design: within-subjects, 3 conditions on the same task.
- Measurement: real diagnostic task with objective scoring — not quiz scores.
- Result: AI-supported training measurably upskilled (and could deskill).
2. Why this is the template
Most programs measure Level 1 (reaction) and Level 2 (learning via self-report). Levels 3 (behavior) and 4 (results) stay empty. The radiology experiment shows how to measure Levels 3-4 with real tasks.
3. Deskilling risk
AI integration can erode competence when not used reflectively. The experiment measured exactly that — the counterpoint to naive "AI makes us better" claims.
4. Transfer to your program
Guide 2 (§5.1) recommends:
- Design: pre/post with comparison group (or within-subjects conditions per task).
- Measurement on real occupational tasks with objective scoring.
- Timeline: baseline (week 0) → training → exit (week 8-12) → behavior follow-up (week 24) → business outcomes (quarter 2-4).
Further reading
Sources: Guide 2 (Evaluation), §3/§5.1; corpus of 16,459 papers (as of 2026-10). EOF echo "en8 done"