AAI-3: AI Systems Specialist

Advanced knowledge of LLM applications, RAG, retrieval, agents, evaluation, monitoring, architecture, and safety.

30 questions ~45 min 80% to pass 70% safety minimum

Someone who passes AAI-3 should be able to:

  • Explain the structure of LLM-powered applications
  • Understand embeddings and vector retrieval
  • Reason about chunking, retrieval, and reranking
  • Design a basic RAG system
  • Design agent workflows and tool boundaries
  • Identify common agent failure modes
  • Create evaluation plans
  • Reason about quality, cost, and latency trade-offs
  • Define guardrails and approval gates
  • Understand production monitoring risks
  • Apply responsible AI and governance principles

Competencies tested: AI Systems, AI Agents, Evaluation and Monitoring, Safety and Responsible AI. See the full competency framework for scoring rules and badge prerequisites.