Full Walkthrough: An AI-Assisted Development Workflow from Requirements to Production

AI Engineer

Introduction to AI Coding Workflow [00:00:15]

Matt Pocock introduces the concept that while AI represents a new paradigm in software development, fundamental software engineering principles remain essential when working with AI. He emphasizes that practices crucial for human collaboration also apply to AI. The workshop aims to guide participants through an AI-assisted development lifecycle, from ambiguous requirements to production features, focusing on these enduring principles.

Understanding LLM Constraints [00:03:00]

The speaker highlights two main constraints when working with Large Language Models (LLMs):

• Smart Zone / Dumb Zone [00:03:06]
  • LLMs perform optimally in a "smart zone" when a conversation begins, as attention relationships are least strained. [00:03:31]
  • Performance degrades into a "dumb zone" as more tokens are added to the context window (scaling quadratically), making the LLM "dumber." [00:04:02] This typically occurs around 100K tokens, regardless of the maximum context window size.

    

  • Implication: Tasks should be sized small to keep the AI within its "smart zone." [00:04:31]
  • Addressing large tasks: Instead of continuously adding to one context (leading to the dumb zone), multi-phase plans break down large tasks into smaller, manageable units, each executed within its own "smart zone" session. [00:05:41] This can be conceptualized as a loop of individual smart zone tasks. [00:06:14]

    

    

• LLMs Forget Context (Memento Effect) [00:07:27]
  • LLMs constantly reset to a base state, forgetting previous interactions unless explicitly maintained. [00:07:31]
  • Session Stages: An LLM session typically involves a tiny system prompt, an exploratory phase, implementation, and testing. [00:07:56]

    

  • Clearing Context: Resets the LLM entirely, going back to just the system prompt. [00:08:42]
  • Compacting Context: Summarizes previous conversations into fewer tokens to preserve context, but the speaker prefers clearing for a consistent, fresh state. [00:10:15] The exact token count is crucial for monitoring proximity to the "dumb zone." [00:09:40]

    

Phase 1: Turning Ambiguous Requirements into a Product Requirements Document (PRD) [00:11:08]

This phase focuses on achieving a shared understanding with the AI before any coding begins. The demo uses the Cadence course management platform.

• The "Grill Me" Skill [00:12:17]
  • The "Grill Me" skill (a small custom prompt) is invoked with a client brief to prevent misalignments between human and AI. [00:15:25]

    

    

  • It forces the AI to interview the user relentlessly, asking clarifying questions one at a time, and offering recommendations based on its exploration of the codebase. [00:16:19]

    

  • Purpose: To reach a "design concept" or shared understanding, much like Frederick P. Brooks' concept, rather than just generating a plan. [01:17:12]
  • Human-in-the-Loop (HITL): This is a HITL task, requiring human interaction to refine ideas and answer AI's questions. [01:26:43]
  • The grilling session can be extensive (e.g., 22 questions answered by AI in the demo), creating a valuable conversation history. [00:20:50]

    

• Generating the PRD [00:30:24]
  • After the grilling session, a write-a-prd skill is used to summarize the refined design concept into a structured Product Requirements Document (PRD). [00:31:19]

    

  • The PRD includes problem statements, solutions, extensive user stories (e.g., 18 in the demo), implementation decisions (data model, modules, services), level thresholds, and testing decisions. [00:34:51]

    

  • Reviewing the PRD: The speaker generally avoids reading the AI-generated PRD in detail, as the alignment was already established during the grilling. He trusts the AI's summarization. [00:35:23]
  • Doc Rot: Keeping static documentation like PRDs in the repository can lead to "doc rot" (outdated documentation), negatively influencing future AI interactions. It's often better to discard them or mark them as closed issues. [01:24:40]

    

    

Phase 2: Breaking Down PRDs into Actionable Tasks (Kanban Board) [00:39:38]

This phase focuses on structuring the work for efficient AI execution.

• Issue with Horizontal Slicing [00:42:50]
  • AI often prefers to code "horizontally" (e.g., completing all database work, then all API work, then all frontend work). [00:43:00]

    

    

  • Problem: This delays feedback, as the full system isn't integrated or testable until later phases. [00:43:22]
• Vertical Slicing (Tracer Bullets) [00:43:45]
  • The "tracer bullet" approach (from The Pragmatic Programmer) involves creating "thin slices of functionality" that cut across all system layers (e.g., database, API, frontend). [00:43:49]

    

  • Benefit: Provides early, integrated feedback at each phase, allowing the AI to test and validate its approach continuously. [00:44:00]
• Kanban Board Generation [00:40:10]
  • The prd-to-issues skill is used to break the PRD into "independently grabbable issues" that represent vertical slices, forming a Kanban board. [00:44:12]

    

  • Each issue specifies its type (AFK or HITL), blocking dependencies, and associated user stories. [00:40:41]

    

  • Parallelization: The Kanban board structure, with explicit blocking relationships, allows multiple AI agents to work on independent tasks simultaneously. [00:50:09]

    

    

  • The speaker manually reviews the generated slices to ensure they are genuinely vertical and not horizontal. [00:45:54]

Phase 3: AI-Assisted Implementation (AFK Agent) [00:53:51]

This is the phase where the human largely steps back, and AI handles the coding.

• AFK Loop for Implementation ("Ralph Loop") [00:53:57]
  • An autonomous coding agent runs in a loop, picking tasks from the Kanban board. [00:54:19]
  • Task Prioritization: The agent prioritizes tasks in a specific order: [00:56:21]
    1. Critical bug fixes
    2. Development infrastructure (tests, types, dev scripts)
    3. Tracer bullets for new features (including "build a tiny, end-to-end slice of the feature first")
    4. Polishing and quick wins
    5. Refactors

       

  • Process: For each task, the agent explores the repo, implements using TDD, runs feedback loops (tests, type checkers), and commits. [00:57:21]
  • Automation: The ralph-once.sh script demonstrates a single iteration of this loop, passing in issues and recent commits. A more complex afk.sh script handles the full looped execution, often within a Docker sandbox. [00:54:24]

    

    

    

• Test-Driven Development (TDD) with AI [01:06:48]
  • Importance: TDD (red, green, refactor) is crucial for AI agents to produce high-quality, reliable code. [01:06:51]
  • Process: The AI first writes a failing test ("red"), then implements the minimal code to make the test pass ("green"), and finally refactors ("refactor"). [01:07:00]
  • Preventing "Cheating": TDD prevents AI from writing "cheating" tests or implementing an entire layer horizontally without proper validation. [01:07:59]
  • Feedback Loops: Robust feedback loops (unit tests, integration tests, type checks, linters) are the "ceiling" for AI coding quality. Improving these loops directly improves the AI's output. [01:09:51]

    

Phase 4: Human Quality Assurance (QA) and Code Review [01:09:12]

Even with autonomous agents, human oversight is non-negotiable.

• Importance of Manual QA: [01:13:21]
  • After an AFK agent completes tasks, humans perform manual QA and code review to inject "taste" and ensure the product meets quality standards. [01:12:47]
  • Automating every step can lead to generic, low-quality "slop." [01:13:02]
  • QA also provides an opportunity to create more issues for the Kanban board, feeding back into the implementation loop. [01:34:53]
• Automated Reviewer [01:05:33]
  • An automated review step can be added after implementation (and before human QA) to check the AI-generated code against coding standards. [01:29:18]
  • The reviewer agent is "pushed" (given) the coding standards and the code to be reviewed, ensuring it operates with a fresh context in its "smart zone." [01:29:23]

    

    

Codebase Design for AI Effectiveness [01:14:10]

The underlying codebase architecture significantly impacts AI's ability to perform well.

• Deep Modules vs. Shallow Modules [01:16:55]
  • Shallow Modules: Characterized by many small files with complex, intertwined dependencies. They are difficult for AI to navigate, understand dependencies, and test effectively. [01:14:42]

    

    

  • Deep Modules: Modules with a small, simple public interface but a large amount of encapsulated, complex functionality inside. [01:17:05]

    

  • Benefit: Deep modules are easier to test (a single test boundary covers much functionality) and easier for AI to work with, leading to better code quality. [01:17:18]
  • AI Tendency: Without explicit guidance, AI tends to produce shallow modules. [01:17:54]
• "Improve Codebase Architecture" Skill [01:21:28]
  • A skill exists to help identify and refactor a codebase towards deep modules. [01:21:37]

    

  • It scans the codebase, identifies architectural coupling and areas for improvement (e.g., services with zero tests, redundant logic), and suggests deepening modules. [01:33:01]
  • Mental Trick: Designing modules with clear interfaces allows developers to delegate the internal implementation details to AI while retaining a high-level understanding of the codebase structure. [01:20:52]

Parallelization with Sandcastle [01:29:50]

For scaling AI-assisted development, parallelization is key.

• Sandcastle Library: A TypeScript library designed to orchestrate AI coding agents in isolated Docker containers (sandboxes), enabling multiple agents to work concurrently. [01:30:07]

  

• Parallel Workflow:
  1. Planner Agent: Reads the Kanban board and selects a set of independent (non-blocking) issues that can be worked on in parallel. [01:31:01]
  2. Implementer Agents: Each selected issue is assigned to a separate implementer agent, running in its own sandbox (Git branch), following the TDD and feedback loop process. [01:31:32]
  3. Reviewer Agents: Optionally, automated reviewer agents can check the implemented code for compliance with coding standards. [01:31:47]
  4. Merger Agent: Once implementations are complete and reviewed, a merger agent integrates the changes from individual branches, resolving any conflicts. [01:31:55]

     

• Push vs. Pull for Coding Standards: For implementers, coding standards can be "pulled" (accessed by the agent as needed via skills). For reviewers, coding standards should be "pushed" (explicitly provided in the prompt) to ensure thorough checking. [01:28:18]