AI World 2 · Grounded Answers · Stage 5

Prove the Retrieval

Retrieve-only or retrieve-and-generate, and the metric that answers the question: coverage, relevance, faithfulness, citations, correctness.

Domain: AI and machine learning
World: AI World 2: Grounded Answers (Associate)
Game type: Solution board: Drop services into each row's slots, then run records through them.
Time: About 25 minutes on a first attempt (estimated)
Exam guides: AIF-C01Task 3.4: Describe methods to evaluate FM performance
MLA-C02Skill 2.3.1: Perform reproducible experiments (for example, by using MLflow on SageMaker AI, Amazon Bedrock evaluations, Amazon Bedrock Prompt Management); Skill 2.3.9: Perform AI evaluation (for example, model output assessment, content quality validation, bias detection, LLM-as-a-judge frameworks); Skill 2.3.10: Configure RAG system monitoring, including retrieval accuracy assessment
AIP-C01Skill 5.1.1: Develop comprehensive assessment frameworks to evaluate the quality and effectiveness of FM outputs beyond traditional ML evaluation approaches (for example, by using metrics for relevance, factual accuracy, consistency, and fluency); Skill 5.1.5: Develop comprehensive assessment systems to ensure thorough evaluation from multiple perspectives for FM outputs (for example, by using RAG evaluation, automated quality assessment with LLM-as-a-Judge techniques, human feedback collection interfaces); Skill 5.1.6: Implement retrieval quality testing to evaluate and optimize information retrieval components for FM augmentation (for example, by using relevance scoring, context matching verification, retrieval latency measurements)