Overview – Advanced Certification in AI Product Management

Topics

• AI-enabled vs AI-native vs agentic products - a working taxonomy
• What breaks in classic PM: specs, QA, estimation, roadmaps, definition of done
• The current AI market map: models, infrastructure, application layer, agent ecosystem
• Where value accrues; wrapper risk and real sources of defensibility
• Build vs buy vs orchestrate decision framework
• Case studies: coding copilots, support deflection, vertical AI (legal, health, finance)
• The AI PM competency model and how the role is actually being hired for

Hands-on / Demo

• Teardown of 3 shipping AI products - archetype, moat and wrapper-risk score
• Build a 1-page AI Product Teardown Canvas for each
• Structured debate: defend one product as defensible, attack another as a thin wrapper

Skills

Topics

• Tokens, embeddings and context windows - what a PM must internalise
• Transformers at intuition level; pretraining vs post-training vs alignment
• LLMs vs classical ML vs deep learning - when each is the right tool
• Reasoning models and test-time compute; when 'thinking' is worth the cost
• Open-weight vs proprietary models; licensing, sovereignty and portability
• Temperature and sampling - why the same prompt returns different answers
• Why hallucination is structural, not a bug waiting to be patched
• The capability-reliability gap and its product implications

Hands-on / Demo

• Run Llama, Qwen, Mistral and Gemma locally via Ollama / LM Studio
• Benchmark 3 models on one identical product task; log quality, latency and cost
• Navigate Hugging Face model cards, licences and leaderboards critically
• Optional Python (async): call an open model from Colab and sweep temperature settings

Skills

Topics

• Supervised, unsupervised and reinforcement learning - what each is good for
• Predictive product families: classification, regression, ranking, recommendation, forecasting, anomaly detection
• The confusion matrix translated into business terms
• False positives vs false negatives - who pays for each, and how much
• Precision, recall, F1 - and when each is the wrong thing to optimise
• Setting the decision threshold as a product decision, not a modelling one
• ROC/AUC and precision-recall curves read as trade-off instruments
• Class imbalance and base rates - why 99% accuracy can be worthless
• Ranking and recommender metrics: relevance, diversity, coverage, cold start
• Model drift, retraining triggers and production monitoring
• When a simple model or plain heuristic beats an LLM on cost, latency and control

Hands-on / Demo

• Business-cost workshop: price false positives and false negatives for a fraud or churn use case, then derive the optimal threshold
• Read a real confusion matrix and precision-recall curve; recommend and defend a threshold to a stakeholder
• Train and tune a no-code classifier in Orange Data Mining or Hugging Face AutoTrain and interpret the evaluation output
• Optional Python (async): scikit-learn notebook plotting business cost against threshold

Skills

Topics

• Jobs-to-be-Done applied to AI-shaped problems
• The AI-fit test: error tolerance, volume, judgement density, verifiability
• Automation vs augmentation vs generation - choosing the intervention level
• Feasibility x value x data-readiness screening
• Sizing ROI: cost displaced, revenue enabled, experience improved
• Opportunity solution trees adapted for probabilistic products
• Anti-patterns: the chatbot reflex, AI-washing, solution-first roadmaps

Hands-on / Demo

• Run the AI Opportunity Scorecard on a real workflow from the learner's own org
• Plot 8 candidate use cases on a feasibility / value matrix

Skills

Topics

• Data strategy for AI PMs; proprietary data as durable advantage
• Data readiness audit: volume, quality, coverage, labelling, freshness
• Labelling strategy, annotation cost and quality control
• Synthetic data - where it helps and where it quietly misleads
• The cold-start problem and bootstrapping approaches
• Feedback loops and designing the data flywheel
• Consent, provenance, licensing and training-data rights
• Data contracts and working agreements with engineering

Hands-on / Demo

• Audit a public dataset (Kaggle / HF Datasets) against a readiness checklist
• Generate synthetic edge-case data with an open model
• Design and present a data flywheel diagram for the Module 4 use case

Skills

Topics

• Prompt anatomy: role, task, constraints, examples, output contract
• Zero-shot, few-shot, chain-of-thought, decomposition, self-critique
• System prompts as behavioural specification - and who owns them
• Structured output, JSON schema and machine-readable contracts
• Context engineering vs prompt engineering; context rot and window budgeting
• Prompt versioning, libraries, testing and ownership across the org
• Designing against prompt injection from the spec stage
• When prompting stops being enough - signals to escalate to RAG or tuning

Hands-on / Demo

• Build a versioned prompt library for a support-triage feature
• Force schema-valid JSON output and handle malformed responses gracefully
• A/B two prompt versions across 20 test inputs and document the winner
• Optional Python (async): batch-run prompts via API in Colab and score outputs

Skills

Topics

• The AI-augmented PM operating system: what to delegate and what never to
• Customer research at scale: transcription, thematic synthesis, quote and sentiment mining
• Synthetic users and AI-simulated interviews - the real limits and failure modes
• Competitive and market analysis with AI research tools; verification discipline
• Mining support tickets, reviews and surveys for opportunity discovery
• Drafting PRDs, user stories and acceptance criteria with AI as first-pass writer
• Spec-to-prototype with coding agents; reviewing code you did not write
• Meeting synthesis, stakeholder updates and executive summaries
• Hallucination hygiene: verification workflows and what never ships unchecked
• Confidentiality: what customer and company data must never enter a prompt

Hands-on / Demo

• Synthesise 10 real user interview transcripts into a themed insight report, then hand-check every theme against source
• Run an AI-assisted competitive teardown, fact-verify each claim and log the error rate
• Generate a PRD first draft from a rough brief, then critique and correct it against a rubric
• Build a personal AI PM workflow playbook of 8-10 reusable, versioned prompts

Skills

Topics

• The prototype-first PM; what evidence actually gets a feature funded
• Vibe coding - what it unlocks and where it collapses
• Interface patterns: conversational, canvas, inline, ambient
• Scoping a demo vs an MVP vs a production feature
• Deployment, sharing and collecting first-user feedback fast
• Prototype hygiene: API keys, cost caps, data handling

Hands-on / Demo

• Build and publicly deploy an AI app with Gradio or Streamlit on Hugging Face Spaces
• Run a 3-user feedback round on the deployed prototype and log findings
• Optional Python (async): Chainlit chat app with streaming responses

Skills

Topics

• Why hallucination happens and what grounding actually fixes
• RAG architecture end to end: ingest, chunk, embed, retrieve, generate
• Chunking and metadata strategy; why document structure matters
• Embeddings and vector stores - the PM-relevant trade-offs
• Citations, provenance and source display as UX primitives
• Knowledge freshness, re-indexing and content ownership
• Long context vs RAG vs both - the cost and quality trade-off
• RAG failure modes: retrieval miss, distractor context, stale source

Hands-on / Demo

• Build a document-grounded assistant with Dify or Flowise plus ChromaDB
• Break it deliberately with 5 adversarial queries and write a failure analysis
• Compare 2 chunking strategies on the same corpus and measure the difference
• Optional Python (async): LangChain / LlamaIndex RAG notebook

Skills

Topics

• Hybrid search: keyword plus semantic, and why pure vectors under-deliver
• Re-ranking and the retrieve-then-rerank pattern
• Query rewriting, expansion and multi-query retrieval
• Graph RAG and structured knowledge for highly connected domains
• Agentic RAG: self-correcting, iterative and multi-hop retrieval
• Permission-aware and multi-tenant retrieval in enterprise deployments
• Retrieval evaluation: context precision, context recall, faithfulness, answer relevance
• The customization decision tree: prompt vs RAG vs fine-tune vs build
• LoRA / QLoRA and distillation - when fine-tuning genuinely wins
• Model routing, cascading and small models for cost control
• Cost, latency and index-maintenance economics at scale

Hands-on / Demo

• Upgrade the Module 9 assistant with hybrid search and a re-ranker; measure the lift
• Build an agentic RAG flow that retries retrieval when confidence is low
• Run a retrieval eval with Ragas and produce a before/after quality report
• Complete a Model Selection Matrix for the learner's use case with defended scoring
• Optional Python (async): LoRA fine-tune a small model with Unsloth and compare against the prompted baseline

Skills

Topics

• Agents vs deterministic workflows - the decision rule
• Tool use and function calling; designing the tool surface
• Model Context Protocol (MCP) and the interoperable tool ecosystem
• Multimodal tool use: vision, document and speech inputs inside agent workflows
• Planning, memory and state; single-agent vs multi-agent patterns
• Autonomy levels and human-in-the-loop checkpoints
• Agent failure modes: loops, drift, runaway cost, silent wrong actions
• Reliability, sandboxing and permissioning for agents that take actions
• Agentic product patterns: copilot to assistant to autonomous co-worker

Hands-on / Demo

• Build a multi-step agent in n8n or Langflow that calls two tools and escalates on low confidence
• Add a document or speech input step using an open vision or Whisper model
• Connect an MCP tool and trace the full execution end to end
• Write the Autonomy and Escalation Specification, then stress-test for loops and cost blowout

Skills

Topics

• Eval-driven development as the core AI PM discipline
• Why public benchmarks mislead product decisions
• Curating golden datasets: coverage, edge cases and adversarial cases
• Rubric design and scoring criteria non-technical raters can apply
• LLM-as-judge: setup, calibration and its known biases
• Human evaluation and annotation workflows
• Offline vs online evaluation; regression testing and prompt CI
• Task success, groundedness, faithfulness and safety metrics
• Closing the loop: eval, diagnosis, improvement, re-eval

Hands-on / Demo

• Curate a 30-case golden dataset for the learner's prototype
• Run automated evals in Promptfoo and Ragas / DeepEval
• Configure an LLM-as-judge rubric and calibrate it against human ratings
• Produce an eval report with an explicit pass/fail release gate

Skills

Topics

• Designing under uncertainty; setting honest expectations at first contact
• Trust calibration, over-reliance and automation complacency
• Confidence, provenance and citation UX
• Progressive disclosure, steerability and user control
• Error recovery, undo and the edit affordance
• Latency UX: streaming, skeletons, optimistic states, background jobs
• Voice and multimodal interaction patterns and their latency budgets
• Feedback capture that genuinely feeds the data flywheel
• Onboarding and mental-model building for non-deterministic features
• AI UX anti-patterns and dark patterns to avoid

Hands-on / Demo

• Critique a live AI feature against a trust-UX heuristic checklist
• Redesign its failure states and produce a Failure-State UX Spec covering wrong, uncertain, refused and slow
• Wireframe the redesign in Penpot and run a peer review

Skills

Topics

• Bias, fairness and representational harm; how to evaluate for them
• Explainability, transparency and disclosure obligations
• EU AI Act risk tiers; NIST AI RMF; ISO/IEC 42001; India DPDP Act implications
• Model cards, system cards and internal AI review boards
• OWASP LLM Top 10 translated for product teams
• Direct and indirect prompt injection; jailbreaks; system-prompt extraction
• Data exfiltration through tools and agents; PII handling, redaction, retention
• Guardrails, content filtering and safe-completion design
• Vendor due diligence, DPAs and model supply-chain provenance
• Red-teaming and AI incident response as recurring product rituals

Hands-on / Demo

• Red-team the learner's own prototype with Garak plus manual injection attempts
• Configure guardrails with Guardrails AI or Llama Guard and evidence the block
• Produce a combined governance and security pack: risk classification, model card, risk register
• Write a prioritised mitigation plan with named owners and target dates

Skills

Topics

• The AI metric stack: task success, containment, deflection, acceptance and edit rate, time-to-value, escalation rate
• Choosing a north star metric for an AI product
• Leading vs lagging quality signals
• Tracing and observability: spans, sessions, token and cost attribution
• A/B testing with stochastic outputs; variance and sample-size traps
• Shadow mode, canary releases and staged rollout
• Guardrail metrics and automatic rollback triggers
• Establishing the weekly AI quality review ritual

Hands-on / Demo

• Instrument the prototype with Langfuse tracing and PostHog product analytics
• Build a metrics dashboard covering quality, cost and adoption
• Write an experiment design doc for one improvement hypothesis, with power and stop criteria

Skills

Topics

• Token economics and AI cost of goods sold
• Gross-margin compression in AI products versus classic SaaS
• Cost drivers: context length, retries, agent loops, reasoning tokens, re-indexing
• Optimisation levers: caching, routing, batching, quantisation, smaller models
• Pricing models: seat, usage, credits, hybrid, outcome-based
• Metering, quotas, fair-use policy and free-tier abuse
• When not to charge for AI; bundling and competitive pricing pressure
• Capacity planning, rate limits and vendor negotiation

Hands-on / Demo

• Build a cost-and-margin model in LibreOffice Calc / Google Sheets for 10K, 100K and 1M monthly requests
• Run a pricing simulation across three models and identify break-even
• Present the recommended pricing model with margin sensitivity analysis

Skills

Topics

• Anatomy of an AI PRD: behaviour spec, eval plan, failure taxonomy, guardrails, rollout
• Writing acceptance criteria for probabilistic outputs
• Spec-driven development and specs as living artifacts
• Roadmapping under model uncertainty; option-based planning
• Org design for AI-native teams; where AI expertise should sit
• Working with data scientists and ML engineers: shared vocabulary and rituals
• Model lifecycle in the roadmap: drift, retraining, deprecation, migration
• Executive communication: translating model risk into business risk
• Managing hype and the 'just add AI' mandate
• Launch readiness review and go / no-go criteria

Hands-on / Demo

• Write a complete AI PRD for the capstone concept, including eval plan and rollout gates
• Run a peer Launch Readiness Review against a scored rubric
• Build a 2-quarter AI roadmap with explicit uncertainty flags
• Deliver a 5-minute executive risk briefing translating one model failure mode into business impact

Skills

Topics

• The pilot-to-production failure data and what it actually indicates
• The four readiness gates: data, integration, economics, governance
• Why the demo works and the deployment does not - integration and systems-of-record debt
• Cost surprise at scale: pilot economics versus production economics
• Defining a production success metric before the pilot starts
• Buy vs build revisited - why internal builds stall at higher rates
• Procurement, security questionnaires, DPAs and the enterprise buying committee
• Pilot fatigue and the organisational cost of repeated stalls
• Change management and adoption when AI changes how people work
• Sunsetting honestly: how and when to kill an AI initiative

Hands-on / Demo

• Run a pilot post-mortem on a real stalled AI initiative and classify root cause against the four gates
• Build a Production Readiness Gate checklist and apply it to the capstone concept
• Draft a pilot charter: success metric, exit criteria, integration plan, production cost estimate
• Mock enterprise review - defend the initiative to a sceptical CFO and security lead played by peers

Skills

Topics

• Capstone brief, artifact set and assessment criteria walkthrough
• Problem framing and archetype selection for the chosen concept
• AI-fit and error-tolerance check before any build begins
• Model / RAG / agent architecture decision with written justification
• Data plan: sources, readiness, licensing and provenance
• Evaluation strategy outlined before the build, not after
• Prototype scoping: where the demo boundary sits versus production
• Backend assembly in Dify, Flowise, Langflow or n8n
• Front end in Gradio or Streamlit
• Deployment to Hugging Face Spaces and public access checks
• Prototype hygiene: API keys, cost caps and data handling

Hands-on / Demo

• Complete the one-page product brief plus architecture diagram and take it through instructor design review
• Assemble the prototype backend and wire in retrieval or tools as the architecture requires
• Deploy to a public URL and verify the prototype runs unaided for a stranger
• Log first-build blockers and clear them in the instructor-supported working session

Skills

Topics

• Golden dataset curation for the capstone concept
• Running the eval suite and reading the results honestly
• Diagnosing and fixing the single largest failure mode
• Guardrail configuration and safe-completion behavior
• Red-team pass and mitigation logging
• Risk classification and the model card
• Tracing instrumentation and the metric set
• Unit-economics and margin model across three volume tiers
• Pricing recommendation tested against the cost curve
• Production readiness gate assessment
• AI PRD completion and artifact pack assembly for submission

Hands-on / Demo

• Curate a golden dataset, run a Promptfoo / Ragas eval and document the pass/fail release gate
• Apply guardrails, complete a red-team pass and log every finding with a mitigation and named owner
• Instrument tracing and build the cost-and-margin model at 10K, 100K and 1M monthly requests
• Assemble and submit the final artifact pack: AI PRD, eval report, risk register, unit-economics model and readiness-gate assessment

Skills