Self-Report Lies: Task-Based AI-Literacy Measurement
The most popular evaluation method in AI training programs is pre/post self-assessment: "How competent do you feel?" It is cheap, fast — and systematically wrong.
1. The divergence (Guide 2, §2.2)
Self-reported and objective measures of AI literacy correlate only weakly (C, K-12 teachers; CFA + latent profile analysis). Programs based on self-report overstate their impact.
Causes:
- LLM fallacy: users attribute AI outputs to their own competence.
- Dunning-Kruger in AI: beginners overestimate, advanced learners underestimate.
- Social desirability: answering "better" after training is socially expected.
2. What to measure instead
Guide 2 (§5.1) recommends a task-based quasi-experiment:
- Design: pre/post with comparison group (or within-subjects conditions per task).
- Measurement on real occupational tasks with objective scoring.
- Instruments: objective literacy scale (AICOS/MAIL-CS) + task performance + SR-vs-OB gap + adoption (UTAUT items) + technostress/well-being.
3. The Kirkpatrick gap
Most programs measure Level 1 (reaction) and Level 2 (learning via self-report). Levels 3 (behavior) and 4 (results) stay empty. That is exactly where the program-evaluation white space lies — and where Art. 4 counts.
Further reading
Sources: Guide 2 (Evaluation), §2.2/§5.1; corpus of 16,459 papers (as of 2026-10). EOF echo "en3 done"