VS Code and GitHub Copilot: Integrated Browser, AI Agent Customization, and Orchestration Updates

Visual Studio Code

Introduction to Rapid Development [0:02:19]

The discussion opens with the acknowledgment of the incredibly fast pace of development in VS Code and GitHub Copilot, where changes are constant and often overwhelming.

• Development speed [0:04:12]
  • Things are moving so quickly that even internal teams rely on automation to keep up.
  • Automation tools, like those for release notes and staying on top of codebase changes, are essential.
  • Example: 200+ Pull Requests (PRs) merged in VS Code and Copilot daily are not manageable manually.
• The role of AI [0:04:57]
  • AI is becoming crucial for managing this complexity, enabling automation beyond human capabilities.
  • The content aims to distill the most important updates using AI to highlight key features from voluminous release notes.

Integrated Browser in VS Code [0:09:00]

A significant enhancement is the new integrated browser, transforming how developers interact with web applications within the IDE.

• Enhanced "Simple Browser" [0:10:01]
  • Reimagined to meet modern front-end development needs, integrating directly with AI agents.
• Functionality
  • Share with Agent [0:10:51]: Allows the AI agent to "see" the current browser view, similar to referencing files in chat.
  • Select Elements [0:11:14]: Users can select specific HTML elements to provide targeted feedback to the agent, capturing screenshots and relevant HTML for context.
  • DevTools Integration [0:12:42]: Directly open browser developer tools within VS Code for inspection and debugging.
  • Agent Navigation [0:17:21]: The agent can programmatically navigate the browser, take screenshots, and execute Playwright-like commands. This saves developer time by automating UI interactions for testing and bug fixing.
  • Artifacts: A forthcoming feature will collect all generated artifacts (e.g., screenshots) in a dedicated tab for easy review, streamlining the feedback loop with agents.

AI Agent Customizations and Debugging [0:23:41]

New tools and features provide deeper control and visibility into how AI agents operate and can be customized.

• Model Picker Improvements [0:15:31]
  • Addressing the rapid increase in available models, the picker now includes search functionality.
  • Presents an "opinionated" set of recommended models based on internal evaluations and experience.
  • Displays context size (e.g., 400k tokens for CodeX) and will soon allow setting "reasoning effort" for models.
• Customization Management Dialogue [0:25:34]
  • A centralized interface to manage all user and workspace customizations (agents, skills, instructions, prompts, hooks).
  • Allows visibility into "extension contributed" agents and skills, helping users understand their origins.
• Agent Debug Panel [0:27:09]
  • Provides a detailed overview of agent sessions, including total tokens burned and tool calls.
  • Session Details [0:29:10]: Displays runtime information, cost (tokens), number of tool calls, and model turns.
  • Logs View [0:29:43]: Trace agent activity throughout its lifecycle, showing performance deltas for loading instructions and skills, and detailed arguments/outputs of tool calls (e.g., read_file, replace_string).
  • Flowchart Visualization [0:30:56]: Visually represents the agent's decision-making flow, offering a clear understanding of its internal processes, with clickable nodes for detailed information.
• Customizable Thinking Phrases [0:36:09]
  • Users can now personalize the messages displayed by AI agents during their "thinking" phase, replacing or appending to default phrases.
  • This allows for injecting humor, team-specific memes, or even motivational messages into the AI's internal monologue.
  • Workspace-level settings (.vscode/settings.json) enable teams to standardize these phrases.

Chat Session Management & UX Improvements [0:46:07]

Improvements focus on making chat interactions more fluid and less visually cluttered.

• Conversation Forking [0:50:46]
  • Allows users to create a new chat session from any point in an existing conversation, preserving relevant context while starting fresh.
  • This is useful for exploring different ideas or refining a specific line of inquiry without losing previous progress.
• Quality of Life UX Tweaks [0:52:12]
  • Contextual Dialogues: Future updates will introduce quick-pick dialogues that appear near the user's cursor, reducing eye movement and improving interaction flow.
  • Input Box Optimization: Ongoing efforts to declutter the chat input box, which currently combines many controls (model picker, context widget, speech, stop button, steering/queuing).
  • Enhanced Title Bar: The chat session title bar can be clicked for quick access to commands, previous sessions, and information.

Measuring Customization Impact with agentRC [0:31:31]

The agentRC tool (formerly "Primer") provides a method for objectively evaluating the effectiveness of AI agent customizations.

• AI Readiness Assessment: Inspects repositories to determine their "AI readiness" and generate instructions/configs.
• Evaluation (agentrc eval) [0:32:23]
  • Uses Copilot SDK to generate a set of test cases for a repository.
  • Compares agent performance (e.g., tool calls, time taken) before and after customizations.
  • Allows developers to objectively assess if a new instruction or skill leads to better outcomes, even if the "resolved rate" remains the same.

Future of AI Orchestration (Sneak Peek) [1:03:07]

The upcoming "Ultralight" stream will delve into advanced orchestration patterns, enabling more autonomous AI agent workflows.

• Orchestrating Local and Cloud Agents [1:03:23]
  • The concept involves a local agent orchestrating multiple cloud coding agents.
  • Cloud agents can submit PRs, which the local orchestrator then pulls, merges, and builds, reducing manual developer overhead.
  • This transforms the developer into a "project lead" who reviews agent work and provides high-level feedback, automating the entire development pipeline.
• Practical Application (Anvil Tool) [1:04:26]
  • Burke demonstrates a project where a local orchestrator manages a 3D town application.
  • The orchestrator processes GitHub issues to generate and integrate features into the town using 3JS, with Copilot ensuring code quality.
  • This showcases how complex workflows can be set up to remove developers further from the direct coding loop, making it more feasible to scale projects.
• Philosophy of AI Development [1:08:05]
  • Emphasizes experimentation and a pragmatic approach: try new ideas, see what works, and don't feel pressured to adopt every new tool immediately.
  • The goal is to explore how AI can augment human creativity and productivity.